Image processing apparatus, image processing method, and program

ABSTRACT

There is provided an image processing apparatus including an association section configured to, in a case where panorama image data generated by using a plurality of frame image data obtained by an imaging operation while displacing an imaging direction is determined to be a full circumference panorama image, associate the panorama image data with information showing that the panorama image data is the full circumference panorama image.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation application of U.S. patentapplication Ser. No. 14/403,858, filed on Nov. 25, 2014, which is anational phase entry under 35 U.S.C. § 371 of International ApplicationNo. PCT/JP2013/060182, filed on Apr. 3, 2013, published on Dec. 12, 2013as WO 2013/183346 A1, which claims priority from Japanese PatentApplication No. JP 2012-129100 filed in the Japanese Patent Office onJun. 6, 2012, the disclosures of which are hereby incorporated herein byreference.

TECHNICAL FIELD

The present disclosure relates to an image processing apparatus, animage processing method, and a program for implementing these, andspecifically relates to an image process appropriate for the display ofpanorama image data.

CITATION LIST Patent Literature

Patent Literature 1: JP 2010-161520A

BACKGROUND ART

As shown in Patent Literature 1, an image process which generates onepanorama image from a plurality of images is well known.

For example, a user obtains a large number of captured images (frameimage data) while horizontally sweeping a camera, and by combiningthese, a so-called panorama image can be obtained.

Note that, a “sweep” is said to be the operation of imaging directionmovement due to a rotational motion of an imaging apparatus at the timeof imaging, in order to obtain a plurality of frame image data forpanorama image generation. For example, in the case of moving an imagingdirection in a horizontal direction, a sweep direction will point to thehorizontal direction.

SUMMARY OF INVENTION Technical Problem

In the case where reproduction and display is performed for a panoramaimage, it is common to perform a so-called single view display or toperform a scroll display.

A single view display is displaying by compressing all panorama imagedata as one still image. In this case, while an entire panorama imagecan be viewed, it will be difficult to become a powerful image due tocompression.

A scroll display is displaying while scrolling by sequentially cuttingout parts of panorama image data. By a scroll display, a panorama imagecan be viewed on a screen larger for each portion.

However, for example, a scroll display is displaying by simplysequentially cutting out panorama image data projected onto acylindrical surface or the like, and a scene from a user viewpoint willnot necessarily be accurately reflected, such as a straight line portionbeing significantly curved such as the horizon or a building, forexample.

On the other hand, there is a reproduction technique in which a highsense of immersion and sense of presence are obtained, by setting apanorama image projected onto a cylindrical surface or the like to adisplay image by re-projecting onto a plane surface screen. Such atechnique of reproduction and display will be called a “projectiondisplay” in the description.

For example, QuickTime VR. An Image-Based Approach to VirtualEnvironment Navigation (Apple Computer, Inc.) or the like is well known.

While it can be considered that the degree of satisfaction of a user whoviews the display of a panorama image can be improved when using such aprojection display, there will be a difficulty in the following point ofsimply performing a projection display instead of a scroll display.

While a viewing angle of a panorama image may be necessary in authoringfor performing a projection display, there is usually little for a userto accurately know the viewing angle of a panorama image projected in apanorama mode of a digital still camera. For example, there a varioustypes of images, such as a full circumference image with a 360° viewingangle, from an image with a viewing angle of approximately 120° or animage with a viewing angle of approximately 180°, in a panorama image.Also, apart from a 360° viewing angle, this calculation for obtaining aviewing angle from panorama image data will become inaccurate due toinfluences such as alignment errors, the sweep radius or distortionaberrations, and an accurate viewing angle will not be able to beobtained.

In this way, a viewing angle will not be known by only panorama imagedata, and authoring of a projection display will not be able to beperformed. For example, to ensure that a viewing angle for a projectiondisplay is known, it may be necessary for a user to roughly estimate aviewing angle at the time of panorama imaging, and to manually input theviewing angle at the time of authoring. This will become a difficultoperation for an ordinary user. Further, when an input viewing angle issignificantly different than an actual one, it will result indistortions occurring in a projection display image.

From this, a projection display will not necessarily always be anoptimum display system from the viewpoint of actual use or from theviewpoint of display quality.

The present disclosure aims to implement an image process in order to beable to manually and accurately execute a display of a panorama image,in particular, a high-quality projection display, for example, by widelyassuming from specialist use or business use up to a device for ageneral consumer.

Solution to Problem

An image processing apparatus according to the present disclosureincludes an association section configured to, in a case where panoramaimage data generated by using a plurality of frame image data obtainedby an imaging operation while displacing an imaging direction isdetermined to be a full circumference panorama image, associate thepanorama image data with information showing that the panorama imagedata is the full circumference panorama image.

An image processing method according to the present disclosure includesperforming a process which, in a case where panorama image datagenerated by using a plurality of frame image data obtained by animaging operation while displacing an imaging direction is determined tobe a full circumference panorama image, associates the panorama imagedata with information showing that the panorama image data is the fullcircumference panorama image.

A program according to the present disclosure causes a calculationprocessing apparatus to execute the process of the image processingmethod.

In such a present disclosure, information in which it is determinedwhether or not it is a full circumference panorama image, such as whenreproduced and displayed, is added to panorama image data combined andgenerated from a plurality of frame image data. That is, in the casewhere at least some panorama image data is determined to be a fullcircumference panorama image, information which shows it is a fullcircumference panorama image is associated with this panorama imagedata. In this way, for example, a display processing system can beselected, according to whether or not it is a full circumferencepanorama image, when performing subsequent reproduction and display.

Here, there will be cases where a full circumference panorama image canbe determined to have a viewing angle of 360° (or approximately 360°).That is, there will be cases where a viewing angle can be fixed, whenperforming a projection display.

Advantageous Effects of Invention

According to the present disclosure, in the case where panorama imagedata is a full circumference panorama image, information which showsthis will be associated. Therefore, it will have an effect in which anappropriate display processing system can be selected, according towhether or not it is a full circumference panorama image, whenperforming reproduction and display. For example, in the case of a fullcircumference panorama image, it becomes possible to execute anappropriate projection display or the like, by setting a viewing angleto 360°.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an explanatory diagram of panorama imaging performed in anembodiment of the present disclosure.

FIG. 2 is an explanatory diagram of panorama image combination performedin an embodiment.

FIG. 3 is an explanatory diagram of a panorama image of an embodiment.

FIG. 4 is an explanatory diagram of a combination projection process anda display projection process of an embodiment.

FIG. 5 is an explanatory diagram of a scroll display and a projectiondisplay of an embodiment.

FIG. 6 is an explanatory diagram of a projection display of a 360°panorama image of an embodiment.

FIG. 7 is an explanatory diagram of a single view display and a listdisplay of an embodiment.

FIG. 8 is a block diagram of an image processing apparatus of anembodiment.

FIG. 9 is a block diagram of configuration examples of an imageprocessing apparatus mounting device of an embodiment.

FIG. 10 is a block diagram of configuration examples of an imageprocessing apparatus mounting device of an embodiment.

FIG. 11 is an explanatory diagram of a first configuration example of anembodiment.

FIG. 12 is a flow chart of processes in the first configuration exampleof an embodiment.

FIG. 13 is an explanatory diagram of a second configuration example ofan embodiment.

FIG. 14 is a flow chart of processes in the second configuration exampleof an embodiment.

FIG. 15 is an explanatory diagram of a third configuration example of anembodiment.

FIG. 16 is a flow chart of processes in the third configuration exampleof an embodiment.

FIG. 17 is an explanatory diagram of a fourth configuration example ofan embodiment.

FIG. 18 is a flow chart of processes in the fourth configuration exampleof an embodiment.

FIG. 19 is a block diagram of an imaging apparatus of a fifthconfiguration example of an embodiment.

FIG. 20 is a block diagram of a functional configuration of a panoramacombination section and an association section installed in the imagingapparatus of an embodiment.

FIG. 21 is an explanatory diagram of panorama images, in the case wherea viewing angle is not 360°, and to which alignment is not performed.

FIG. 22 is an explanatory diagram of a panorama image to which alignmentis performed of an embodiment.

FIG. 23 is an explanatory diagram of alignment of a panorama image of anembodiment.

FIG. 24 is a flow chart of processes at the time of panorama imaging ofan embodiment.

FIG. 25 is an explanatory diagram of display mode transition of anembodiment.

FIG. 26 is a block diagram of a functional configuration of a displayimage generation section installed in the imaging apparatus of anembodiment.

FIG. 27 is a flow chart of a display process of an embodiment.

FIG. 28 is a block diagram of a computer apparatus of an embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, the embodiments will be described in the following order.

Note that, a device having at least the association section 11 in theembodiments will become the image processing apparatus in the claims.

<1. Outline of panorama combination and reproduction display>

<2. Image processing apparatus configuration examples>

<3. First configuration example (metadata)>

<4. Second configuration example (dedicated program)>

<5. Third configuration example (file link information)>

<6. Fourth configuration example (dedicated format)>

<7. Fifth configuration example (application example to an imagingapparatus)>

(7-1: Configuration of the imaging apparatus)

(7-2: Panorama combination and association processes)

(7-3: Panorama image display)

<8. Sixth configuration example (application example to a computerapparatus and program)>

<9. Modified example>

1. Outline of Panorama Combination and Reproduction Display

First, an outline of panorama combination will be described.

An imaging apparatus 50 of an embodiment, which will be described later,or a general imaging apparatus of recent years (a digital still camera),can generate a panorama image by performing a combination process for aplurality of still images (frame image data) obtained by a photographerwho is performing imaging while rotatably moving the imaging apparatus50 around some rotation axis.

In particular, a full circumference panorama image with a viewing angleof 360° can be generated by the imaging apparatus 50 of an embodiment.

FIG. 1A shows the movement of the imaging apparatus 50 at the time ofpanorama imaging. Since unnaturalness of junctions will occur whenparallaxes of the background and foreground are combined at the time ofpanorama imaging, it is desirable for a rotation center at the time ofimaging to be set to a rotation center of a point specific to a lens, atwhich parallax called nodal point (Nodal Point) is does not occur.Rotational movement of the imaging apparatus 50 at the time of panoramaimaging will be called a “sweep”.

FIG. 1B is a schematic diagram at the time when an appropriate alignmentis performed for a plurality of still images obtained by a sweep of theimaging apparatus 50. In this figure, each still image obtained byimaging is shown in a time sequence of imaging. That is, frame imagedata captured from a time 0 up to a time (n−1) is set to frame imagedata FM #0, FM #1 . . . FM #(n−1).

In the case where a panorama image is generated from n still images, acombination process is performed for a series of n frame image data FM#0 to FM #(n−1) continuously captured such as that illustrated.

As shown in this FIG. 1B, since it may be necessary for each capturedframe image data to have overlapping portions with adjacent frame imagedata, it is necessary for an imaging time interval of each frame imagedata of the imaging apparatus 50, and an upper-limit value of a speed atwhich a photographer performs a sweep, to be appropriately set.

Also, since a frame image data group aligned in this way will have manyoverlapping portions, it may be necessary to determine a region to beused for a final panorama image for each frame image data. In otherwords, this is the same as determining a junction (seam: seam) of imagesin a panorama combination process.

In FIG. 2A and FIG. 2B, examples of seams SM are shown.

In a seam, it can be set to a vertical straight line in a sweepdirection such as shown in FIG. 2A, or a non-straight line (curved lineor the like) such as shown in FIG. 2B.

In this FIG. 2A and FIG. 2B, a seam SM0 is shown as a junction betweenframe image data FM #0 and FM #1, a seam SM1 is shown as a junctionbetween frame image data FM #1 and FM #2, . . . and a seam SM(n−2) isshown as a junction between frame image data FM #(n−2) and FM #(n−1).

Note that, by having these seams SM0 to SM(n−2) set to junctions betweenadjacent images at the time of combining, the portions which have beenmade shaded parts in each frame image data will become image regionswhich are not used for a final panorama image.

Further, when performing panorama combination, an objective is set forreducing unnaturalness of images in the vicinity of the seams, and therewill be cases where blend processing is also performed for image regionsin the vicinity of the seams. Also, there will be cases where commonportions of each frame image data are connected by blend processing in awide range, or there will be cases where pixels contributing to thepanorama image are selected for each pixel from the common portions, andin these cases, while junctions will clearly be present, these widerange connection portions will have the same appearance as seams for thepresent disclosure.

Further, as shown in FIG. 1B, as a result of the alignment of each frameimage data, some movement will be recognized not only in the generalsweep direction, but also in a direction perpendicular to that of thesweep. There will be gaps occurring by this due to camera shake of aphotographer at the time of the sweep.

By determining the seams of each frame image data, performingconnections by connection or blend processing at these boundary regions,and finally trimming unnecessary portions in a direction perpendicularto that of a sweep by considering a camera shake amount, a panoramaimage with a wide viewing angle can be obtained, with a sweep directionset to a long side direction, such as shown in FIG. 3.

In FIG. 3, vertical lines show seams, and a state is schematically shownin which a panorama image is generated by respectively connecting nframe image data FM #0 to FM #(n−1) at seams SM0 to SM(n−2).

A specific operation example of panorama imaging, which is performed bya user for obtaining such a panorama image by using the imagingapparatus 50, will be described.

First to begin with, the imaging apparatus 50 is set to a panoramaimaging mode, and a maximum image viewing angle is additionally set. Forexample, 120°, 180°, 360° or the like is capable of being set as themaximum image viewing angle, and in the case where 360° is selected,so-called full circumference imaging can be performed which stores anentire scene of the surroundings for all photographs. Note that, anexpression such as “360° panorama imaging mode” is used in the casewhere the maximum image viewing angle is set to 360° in the panoramaimaging mode, and an expression such as “180° panorama imaging mode” isused in the case where the maximum image viewing angle is set to 180°.

Next, panorama imaging is started by having a photographer press ashutter button of the imaging apparatus 50, and a sweep operation isperformed. Afterwards, in the case where end conditions of panoramaimaging are satisfied, the panorama imaging will be completed. Forexample, the end conditions are as follows.

-   -   The case of reaching a maximum image viewing angle set in        advance.    -   The case of the sweep operation being stopped by a user.    -   The case of a user pressing the shutter button again.    -   The case of a user removing a finger from the shutter button        (for a specification of continuing to press the shutter button        during panorama imaging).    -   The case of some error occurring.

After imaging is completed, a panorama combination process isautomatically started, a combination of a panorama image is completedafter a certain time has elapsed, and this panorama image data is heldin a recording device. Afterwards, a list is usually made the same asthat of the still images, and can be displayed.

Generated panorama image data such as this and a display mode of thiswill be described.

FIG. 4A is a schematic diagram of a projection process at the time ofpanorama combination. While the processing systems at the time ofpanorama combination are diverse, a process is often performed, forexample, which projects onto a cylindrical surface, a spherical surface,a cube or the like. This will be called a “combination projectionprocess” in the description, and is distinguished from a “displayprojection process” which is performed for a projection display, whichwill be described later.

In this FIG. 4A, a large amount of frame image data FM, which iscaptured while sweeping at the time of panorama imaging, is shown. Itcan be said that frame image data FM, that is, a captured image the sameas that of normal photographing, projects a three-dimensional scene ontoa plane surface (for example, an imager plane surface of a camera). Whenpanorama image data such as that of FIG. 3 is generated from such alarge amount of frame image data FM, a state will be set in which eachframe image data FM is projected onto a cylindrical surface FR, forexample, as a combination projection process. In this diagram, projectedframe image data FMpd is schematically shown along a cylindrical surfaceFR.

Panorama image data PD such as that of FIG. 4B is generated byconnecting the frame image data FMpd to which such a combinationprojection process has been performed. The panorama image data PD can beimage data to which projection has been performed again on thecylindrical surface FR as seen from a projection center CT.

In the present embodiment, a single view display, a list display, ascroll display and a projection display are capable of being executed asdisplay modes for such panorama image data PD. The display modes areillustrated in FIG. 5, FIG. 6 and FIG. 7.

FIG. 5 schematically shows the concepts of a scroll display and aprojection display.

A scroll display cuts out parts of the panorama image data PD, which isan image projected onto a cylindrical surface or the like such as thatdescribed above, as they are, and displays by scrolling up, down, leftand right automatically or in accordance with a manual operation.

On the other hand, a projection display is different to that of a scrolldisplay, and displays an image to which projection (a display projectionprocess) is performed again onto a virtual plane surface screen VS forpanorama image data PD projected onto a cylindrical surface, a sphericalsurface, cube or the like.

That is, in this case, a process is performed which projects, onto aplane surface, panorama image data projected onto a projection surfaceat the time of panorama image generation as a display projectionprocess. Also, an image projected onto a plane surface is set to displayimage data.

In this way, an image at the time of imaging can be reproduced,projection distortions caused by cylindrical projection or sphericalsurface projection as seen from a scroll display are eliminated, and adisplay with an improved sense of immersion and sense of presence isachieved.

For example, FIG. 6 schematically shows a projection display of panoramaimage data PD obtained in a 360° panorama imaging mode. Panorama imagedata PD is image data projected onto a 360° cylindrical surface FR ofthe surroundings of a projection center CT. This panorama image data PDis projected onto virtual plane surface screens VSO . . . VS45 . . . orthe like corresponding to each angle. For example, the plane surfacescreen VSO is shown as a plane surface screen centered on a 0° directionfrom the projection center CT, and the plane surface screen VS45 isshown as a plane surface screen centered on a 45° direction from theprojection center CT. In the figure, each of the plane surface screensVS135, VS180, VS225 and VS315 of the 135° direction, the 180° direction,the 225° direction and the 315° direction are additionally illustrated.

In a projection display, an image projected onto the plane surfacescreen VS set to such a viewing direction from the projection center CTis displayed. By appropriately setting a vector or distance from theprojection center CT to the plane surface screen VS automatically or inaccordance with a manual operation, a change of viewing direction or azoom-in/zoom-out are possible.

For example, if a projection image of the plane surface screen VSO, andprojection images of the plane surface screen VS1 (1° direction), theplane surface screen VS2 (2° direction) . . . the plane surface screenVS45 . . . the plane surface screen VS135 . . . the plane surface screenVS315 . . . and the plane surface screen VS0 are sequentially displayed,it becomes possible to display so as to look over the full circumferencethe surroundings from the projection center CT.

Further, the display image will be zoomed-in/zoomed-out, by moving theposition of the plane surface screen VS far from the projection centerCT or bringing it close to the projection center CT.

However, apart from a full circumference panorama image captured in a360° panorama imaging mode, a viewing angle of panorama image data willnot be able to be accurately obtained. Also, a three-dimensional modelat the time of performing a display projection process will not able tobe strictly defined. Accordingly, that other than 360° panorama imagedata will not be appropriate for a projection display.

Next, FIG. 7A shows an example of a single view display, and FIG. 7Bshows an example of a list display.

Since a single view display generates display image data which simplyincludes all the panorama image data PD, and displays this, it is anormal display operation. For example, by having a panorama image becomea horizontally long image, and combining it with a different aspectratio of a display screen, parts on the screen such as shown in FIG. 7A(in the case of this example, an upper region and a lower region) willoften become regions which are not displayed.

As shown in FIG. 7B, a list display displays image data capable of beingreproduced and displayed in a list. For example, by having a userperform image specification in a list display, panorama image data PD towhich the above described scroll display or projection process isperformed can be selected.

Note that, for example, while the images of the list display of FIG. 7Bare not images in which entire horizontally long panorama images arecompressed, in the case where panorama image data PD is represented in alist display, parts of a panorama image may be cut out and compressed,and a display may be performed such as that illustrated. It goes withoutsaying, for example, that there will also be list display modes such asa plurality side by side, by compressing the entire horizontally longpanorama images.

2. Image Processing Apparatus Configuration Examples

Configuration examples of an image processing apparatus of an embodimentwill be described.

FIG. 8A shows a configuration example having an image processingapparatus 1, which includes an association section 11 for generatedpanorama image data, and a panorama combination section 10. Theassociation section 11 and the panorama combination section 10 may beincluded in one body within a same device, or may be each included indifferent device bodies.

Further, FIG. 8B shows a configuration example having an imageprocessing apparatus 2, which includes a display image generationsection 20 for displaying panorama image data, and a display section 21.The display image generation section 20 and the display section 21 maybe included in one body within a same device, or may be each included indifferent device bodies.

In the configuration of FIG. 8A, the panorama combination section 10generates panorama image data PD from an input image group FMS. Theinput image group FMS is a series of frame image data FM obtained byhaving a photographer perform an imaging operation while sweeping in apanorama imaging mode. The panorama combination section 10 generatespanorama image data PD such as that described above in FIG. 1 to FIG. 3,by using a plurality of frame image data FM as the input image groupFMS.

Then, the panorama combination section 10 supplies the generatedpanorama image data PD to the association section 11. Note that, adetailed processing example of the panorama combination section 10 willbe stated in the description of a fifth configuration example of anembodiment.

Further, the panorama combination section 10 also supplies various typesof panorama processing information Ip to the association section 11 inthe generation process of the panorama image data PD.

Panorama processing information Ip is the generic name for the varioustypes of information generated in the processes of the panoramacombination process.

For example, information which shows a mode type such as a 360° panoramaimaging mode or a 180° panorama imaging mode (hereinafter, panorama modetype information) can be considered as the content of the panoramaprocessing information Ip.

Further, information which shows a determination result of whether ornot the panorama image data PD has a 360° viewing angle (hereinafter,360° viewing angle determination information) can be considered as thecontent of the panorama processing information Ip. For example, it isinformation of a result in which it is determined whether or not it isan exactly 360° viewing angle. Note that, it may be a result in which itis determined to have a roughly approximate 360° viewing angle.

Further, information which shows whether or not the panorama image dataPD has a 360° viewing angle, and that a both end alignment is performed(hereinafter, both end alignment information) can be considered as thecontent of the panorama processing information Ip. While both endalignment will be stated in the description of a fifth configurationexample of an embodiment, it can be simply said that it is a processwhich performs an adjustment so that there are no image gaps in thevertical direction and the horizontal direction at the 0° and 360°positions of the panorama image, that is, at both ends of the panoramaimage.

Further, information of a projection surface of a combination projectionprocess (hereinafter, combination projection surface information) can beconsidered as the content of the panorama processing information Ip. Forexample, it is information which shows a cylindrical surface, aspherical surface, a polygonal surface, a plane surface or the like.

Further, information which shows a horizontal viewing angle and avertical viewing angle of the panorama image data PD (hereinafter,horizontal and vertical viewing angle information) can be considered asthe content of the panorama processing information.

In the case where the generated panorama image data PD is determined tobe a full circumference panorama image, the association section 11performs a process (association process) which associates this panoramaimage data PD with information with shows that this panorama image dataPD is a full circumference panorama image. Specific process examples, asthe association process, will be described in a first configurationexample to a fourth configuration example of an embodiment.

Also, the association section 11 outputs panorama image data forrecording or transmitting. In order for a distinction in thedescription, panorama image data for recording or transmitting, which isoutput by the association section 11, will be called, in particular, “apanorama image file PDr”.

This panorama image file PDr is panorama image data which is recorded toa recording medium or is transmitted to another device. In particular,in the case where the panorama image data PD has an approximately 360°viewing angle (or an exactly 360° viewing angle), there will be one or aplurality of data files of a state in which information which shows itis a full circumference panorama image is associated.

To continue, in the configuration of FIG. 8B, the display imagegeneration section 20 obtains the panorama image file PDr.

For example, the panorama image file PDr recorded to a recording mediumby being output from the above described association section 11 issupplied to the display image generation section 20 by being read fromthe recording medium. Alternatively, the panorama image file PDrtransmitted and output by being output from the association section 11is supplied to the display image generation section 20 by beingreceived.

For example, the display image generation section 20 generates displayimage data PDdsp for the above described single view display, listdisplay, scroll display or projection display. In particular, in thecase where a panorama image of the panorama image file PDr selected as adisplay target is determined to be a full circumference panorama image,display image data PDdsp for a projection display is generated byperforming a display projection process.

Also, the display image generation section 20 supplies the display imagedata PDdsp to the display section 21. Specific process examples of thedisplay image generation section 20 will be described in the firstconfiguration example to the fourth configuration example of anembodiment.

The display section 21 has display panel such as a liquid crystal panel,an organic Electroluminescence (EL) panel, a plasma display panel or aCathode Ray Tube (CRT), and a display drive section for this displaypanel, and performs various types of display. By having the displayimage data PDdsp supplied from the display image generation section 20a, a display operation based on this display image data PDdsp isexecuted in this display section 21. For example, a single view display,a list display, a scroll display or a projection display related to apanorama image is performed.

In the case of the present example, in the case where a panorama imageof the panorama image file PDr selected as a display target isdetermined to be a full circumference panorama image, display image dataPDdsp is generated by performing a display projection process by thedisplay image generation section 20, and is supplied to the displaysection 21. For example, images projected onto the virtual plane surfacescreen VS of FIG. 6 are sequentially supplied, and in this case, aprojection display is executed by the display section 21.

Various types of configuration examples can be considered as anelectronic device which has the image processing apparatus 1, whichincludes the association section 11, and the image processing apparatus2, which includes the display image generation section 20, such asdescribed above. Various types of apparatus configuration examples willbe illustrated in FIG. 9 and FIG. 10 as the electronic device.

FIG. 9A is an example in which the image processing apparatus 1 and theimage processing apparatus 2 are constituted in an apparatus DVC1 whichincludes an imaging function.

The apparatus DVC1 of this example has an imaging section 12, a panoramacombination section 10, an association section 11, a recording andreproduction section 13, a display image generation section 20, and adisplay section 21.

For example, the imaging section 12 obtains a series of a plurality offrame image data FM for generating a panorama image, by an imagingoperation of panorama image data. This plurality of frame image data FMis supplied to the panorama combination section 10 as an input imagegroup FMS. The panorama combination section 10 generates panorama imagedata PD from the input image group FMS, and outputs panorama image dataPD and panorama processing information Ip. The association section 11performs the above described association process, and outputs a panoramaimage file PDr. The panorama image file PDr is recorded to a recordingmedium in the recording and reproduction section 13.

The panorama image file PDr read from the recording medium in therecording and reproduction section 13 is supplied to the display imagegeneration section 20, and display image data PDdsp is generated. Thisdisplay image data PDdsp is supplied to the display section 21, and adisplay such as a projection display is executed.

In this way, for example, the apparatus DVC1 can be assumed to be adigital still camera, a video camera, a mobile phone unit, a computerapparatus or the like, which includes a panorama imaging function, arecording and reproduction function, and a display function.

FIG. 9B is an example in which the image processing apparatus 1 and theimage processing apparatus 2 are constituted in an apparatus DVC2 whichincludes an image data input function from the outside.

The apparatus DVC2 of this example has an input section 14, a panoramacombination section 10, an association section 11, a recording andreproduction section 13, a display image generation section 20, and adisplay section 21. That is, when compared to the above described FIG.9A, there will be cases where the input section 14 is included insteadof the imaging section 12.

For example, the input section 14 inputs a series of a plurality offrame image data FM for generating a panorama image, which is obtainedby an imaging operation of panorama image data in an external imagingapparatus. This plurality of frame image data FM is supplied to thepanorama combination section 10 as an input image group FMS. Theoperations of each section hereinafter are the same as those of FIG. 9A.

In this way, the apparatus DVC2 is a device which performs panoramaimage generation by using frame image data FM obtained in an externalimaging apparatus, or performs a display of a panorama image. Forexample, it can be assumed to be an image editing device, a recordingand reproduction apparatus, a television reception device, a mobilephone unit, a computer apparatus or the like used by wired or wirelesslyconnecting to an imaging apparatus such as a digital still camera.

An apparatus DV3 of FIG. 9C is an example in which the image processingapparatus 1 and the image processing apparatus 2 are constituted as adevice which does not have the display section 21.

The Apparatus DVC3 of this example has an input section 14 (or animaging section 12), a panorama combination section 10, an associationsection 11, a recording and reproduction section 13, and a display imagegeneration section 20. The display section 21 of this case becomes anexternal device connected to the apparatus DVC3. While the operations ofeach section are the same as those of FIG. 9A and FIG. 9B, the displayimage generation section 20 transmits the generated display image dataPDdsp to the external display section 21, and causes a display such as aprojection display to be executed.

In this way, the image processing apparatus 1 and the image processingapparatus 2 are capable of being executed as a device which does nothave a display function.

FIG. 10A is an example of an apparatus DVC4 which becomes the imageprocessing apparatus 1, and an apparatus DVC5 which becomes the imageprocessing apparatus 2.

The apparatus DVC4 has an imaging section 12 (or an input section 14), apanorama combination section 10, an association section 11, and arecording section 13R.

The recording section 13R records a panorama image file PDr from theassociation section 11 to a portable recording medium 90. It can beassumed to be a memory card, an optical disc, a magnetic disc or thelike as the portable recording medium 90.

The apparatus DVC5 has a reproduction section 13P, a display imagegeneration section 20, and a display section 21.

The reproduction section 13P reproduces a panorama image file PDr fromthe portable recording medium 90, and supplies it to the display imagegeneration section 20. The display image generation section 20 generatesdisplay image data PDdsp based on the panorama image file PDr, andcauses a display of a panorama image to be executed by supplying to thedisplay section 21.

In such an example, a configuration in which a panorama image file PDris delivered by the portable recording medium 90 can be considered as aconfiguration in which the apparatus DVC4 which becomes the imageprocessing apparatus 1 and the apparatus DVC5 which becomes the imageprocessing apparatus 2 are different bodies. Note that, an example inwhich the external display section 21 is used can also be considered,which does not include the display section 21 in the apparatus DVC5.

FIG. 10B is an example of an apparatus DVC6 which becomes the imageprocessing apparatus 1, and an apparatus DVC7 which becomes the imageprocessing apparatus 2.

The apparatus DVC6 has an imaging section 12 (or an input section 14), apanorama combination section 10, an association section 11, and acommunication section 15.

The communication section 15 transmits a panorama image file PDr fromthe association section 11 to the outside.

The apparatus DVC7 has a communication section 16, a display imagegeneration section 20, and a display section 21.

The communication section 16 receives the panorama image file PDrtransmitted from the apparatus DVC6, and supplies it to the displayimage generation section 20. The display image generation section 20generates display image data PDdsp based on the panorama image file PDr,and causes a display of a panorama image to be executed by supplying tothe display section 21.

In such an example, a configuration in which a panorama image file PDris delivered by communication can also be considered as a configurationin which the DVC6 which becomes the image processing apparatus 1 and theDVC7 which becomes the image processing apparatus 2 are differentbodies.

Communication between the communication sections 15 and 16 can also beconsidered for either wired communication or wireless communication.Further, it will be assumed that the communication is a so-calledcommunication mode, such as communication by a cable connection betweendevices, communication by a Universal Serial Bus (USB) or the like,communication by a Local Area Network (LAN), or communication using apublic network such as the internet. In addition, the communicationsection 15 may perform communication which uploads a panorama image filePDr to a server or the like on the internet, or the communicationsection 16 may perform communication which downloads a panorama imagefile PDr from a server. Further, the communication sections 15 and 16may perform communication in which cloud computing is assumed.

Note that, an example in which the external display section 21 is usedcan also be considered, which does not include the display section 21 inthe apparatus DVC7.

Heretofore, while device configuration examples have been shown, thedevice configurations which implement the image processing apparatuses 1and 2 can be additionally considered for various types.

For example, in the case where a cloud computing system is assumed, theassociation section 11 and the display image generation section 20 canalso be considered to be arranged on a network and not in user usedevices.

Further, the above described panorama image file PDr is data to which aprocess is performed which associates panorama image data PD withinformation which shows that this panorama image data PD is a fullcircumference panorama image. While it can be considered that a file ismade in which information which shows that it is a full circumferencepanorama image (for example, metadata or a dedicated program, which willbe described later) is incorporated with the panorama image data PD asthis panorama image file PDr, additional panorama image data PD andinformation which shows that it is a full circumference panorama imagecan be included as different files, and there will also be a mode inwhich these are associated by some information.

3. First Configuration Example (Metadata)

Hereinafter, specific configuration examples as the association section11 and the display image generation section 20 shown in FIG. 8 will bedescribed. A first configuration example is an example in which theassociation section 11 performs an association process using metadata.

FIG. 11A shows the panorama combination section 10 and the associationsection 11 in the image processing apparatus 1. In this firstconfiguration example, the association section 11 has a metadatageneration section 11 a and a data integration section 11 b.

The panorama combination section 10 generates panorama image data PD andoutputs panorama processing information Ip.

The panorama processing information Ip is supplied to the metadatageneration section 11 a. The metadata generation section 11 a generatesmetadata MT reflecting the panorama processing information Ip. There ispanorama mode type information, 360° viewing angle determinationinformation, both end alignment information, combination projectionsurface information, horizontal and vertical viewing angle informationor the like as the content of the above described panorama processinginformation Ip.

In the case where the generated panorama image data PD is determined tobe a full circumference panorama image, the panorama mode typeinformation, the 360° viewing angle determination information or theboth end alignment information from among these can become informationwhich shows that this panorama image data PD is a full circumferencepanorama image.

The data integration section 11 b generates a panorama image file PDr byintegrating the panorama image data PD generated by the panoramacombination section 10 with the metadata MT generated by the metadatageneration section 11 a. For example, the panorama image file PDrbecomes a structure such as that of FIG. 11B. That is, it is a data filestructure which includes a header, panorama image data PD, and metadataMT.

In each of the apparatuses DVC1 to DVC3, DVC4 and DVC6 illustrated inFIG. 9 and FIG. 10, such a panorama image file PDr is recorded to arecording medium or is transmitted and output.

FIG. 11C shows the display image generation section 20 in the imageprocessing apparatus 2. In this first configuration example, the displayimage generation section 20 has a data separation section 20 a, ametadata interpretation section 20 b, and an image generation processingsection 20 c.

The panorama image file PDr, which is reproduced from the recordingmedium or is received, is input to the data separation section 20 a. Thedata separation section 20 a separates the metadata MT and the panoramaimage data PD from this panorama image file PDr.

The metadata MT is interpreted by the metadata interpretation section 20b, and is supplied to the image generation processing section 20 c aspanorama processing information Ip (panorama mode type information, 360°viewing angle determination information, both end alignment information,combination projection information, horizontal and vertical viewingangle information).

The image generation processing section 20 c generates and outputsdisplay image data PDdsp based on the panorama image data PD. At thistime, it is determined whether or not to perform display image dataPDdsp generation for a projection display, in accordance with thecontent of the panorama processing information Ip. That is, in the casewhere the panorama image data PD of a display target is determined to bea full circumference panorama image, a process for a projection displayis performed, and in the case where it is not a full circumferencepanorama image, a process for a scroll display or a single view display,for example, is performed.

Then, the generated display image data PDdsp is supplied to a displaysection which is not illustrated, or a device having a display section,and a display is executed.

In each of the apparatuses DVC1 to DVC3, DVC5 and DVC7 illustrated inFIG. 9 and FIG. 10, this display image data PDdsp is supplied to thedisplay section 21 and a panorama image display is performed.

Process examples of the above described image processing apparatus 1 andthe image processing apparatus 2 are shown in FIG. 12.

FIG. 12A shows the processes of the panorama combination section 10 andthe association section 11 as the image processing apparatus 1.

In step F1, the panorama combination section 10 obtains an input imagegroup FMS.

In step F2, panorama image data PD is generated by having the panoramacombination section 10 perform a panorama combination process. Thepanorama combination section 10 supplies panorama processing informationIp obtained in this panorama combination processing process to theassociation section 11 (the metadata generation section 11 a).

In step F3, the association section 11 (the metadata generation section11 a) generates metadata MT based on the panorama processing informationIp.

In step F4, the association section 11 (the data integration section 11b) integrates the panorama image data PD with the metadata MT, andgenerates, for example, a panorama image file PDr such as that of FIG.11B.

In step F5, the panorama image file PDr is recorded to a recordingmedium by a recording and reproduction section which is not illustrated,or is transmitted and output to an external device, a network or thelike by a communication section which is not illustrated.

In FIG. 12B, the processes of the display image generation section 20 ofthe image processing apparatus 2 are shown.

In step F51, the display image generation section 20 (the dataseparation section 20 a) obtains the panorama image file PDr, which isreproduced by the recording medium which is not illustrated, or isreceived by the communication section which is not illustrated.

In step F52, the display image generation section 20 (the dataseparation section 20 a, the metadata interpretation section 20 b)separates the panorama image data PD and the metadata MT from thepanorama image file PDr, extracts the metadata MT, and obtains panoramaprocessing information Ip by interpreting this.

In step F53, the display image generation section 20 (the imagegeneration processing section 20 c) branches the processes by whether ornot the panorama image data PD of a display target of this time is afull circumference panorama image with a 360° viewing angle.

The determination of whether or not the panorama image data PD is a fullcircumference panorama image can be considered for various types.

By referring to 360° viewing angle determination information in thepanorama processing information Ip, it can be determined whether or notthe panorama image data PD has an exactly 360° viewing angle.Accordingly, it may be determined to be a full circumference panoramaimage in the case where it has an exactly 360° viewing angle.

Further, by referring to both end alignment information in the panoramaprocessing information Ip, it can be determined whether or not thepanorama image data PD has a 360° viewing angle, and a both endalignment process is performed. Accordingly, it may be determined to bea full circumference panorama image by setting, as a condition, having a360° viewing angle and both end alignment being performed.

Further, by referring to panorama mode type information in the panoramaprocessing information Ip, it can be determined whether or not panoramaimage data PD is captured in a is captured in a 360° panorama imagingmode, that is, whether or not it has an approximately 360° viewingangle. Accordingly, it may be determined to be a full circumferencepanorama image if it is captured in a 360° panorama imaging mode.

Note that, it may not necessarily be said that the panorama image dataPD captured in a 360° panorama imaging mode has an exactly 360° viewingangle. This is because, in a 360° panorama imaging mode, while panoramaimage data PD is generated from frame image data FM obtained by a 360°sweep at the time of imaging, the sweep is performed by a user, and willnot necessarily become an exactly 360° viewing angle. There will becases where the user stops the sweep before reaching 360°. Therefore,the panorama image data PD captured in a 360° panorama imaging modebecomes an image estimated to have an approximately 360° viewing angle.

In step F53, any of the above described determination techniques areused, and in the case where it is determined to be a full circumferencepanorama image, the display image generation section 20 (the imagegeneration processing section 20 c) proceeds to step F54, generatesdisplay image data PDdsp by performing a display projection process, andin step F55, performs a process which outputs this display image dataPDdsp. The processes of these steps F54 and F55 are continued up to itbecomes a display end, in step F56, by a user operation or an automaticprocess. In this way, in the display section, for example, a display soas to look over the full circumference or a display of an arbitrarydirection is executed as a projection display.

Here, in a display projection process for performing a projectiondisplay, information of a viewing angle of a panorama image may benecessary. In the case of the present example, the panorama image dataPD which becomes the process target of step F54 is determined to be afull circumference panorama image such as described above, andaccordingly, the image generation processing section 20 c may perform adisplay projection process by setting the panorama image data PD to aviewing angle of 360°. In this way, a high-quality projection display inwhich there are few image distortions can be performed.

Further, in a display projection process, panorama image data PD, towhich projection is performed onto a projection surface (cylindricalsurface, spherical surface, polygonal surface or the like) at the timeof panorama combination, is re-projected onto a virtual plane surfacescreen VS. Accordingly, information of the projection surface of thepanorama image data PD may become necessary. In the present example, byreferring to combination projection surface information in the panoramaprocessing information Ip, the image generation processing section 20 ccan accurately perform a re-projection process to the plane surfacescreen VS.

Further, in a projection display, by changing a horizontal and verticalviewing angle of the virtual plane surface screen VS, a display can beperformed which looks over the surroundings in the horizontal andvertical directions the same as that of a scroll display. For example,by having a 360° viewing angle in the horizontal direction, whileunrestricted scrolling can be performed in the horizontal direction, itwill be restricted in the vertical direction. In this case, by referringto horizontal and vertical viewing angle information in the panoramaprocessing information Ip, the image generation processing section 20 ccan restrict a scroll capable range of the vertical direction.

Note that, horizontal and vertical viewing angle information may beinformation which is simply used for a restriction of a display range,without being used in the projection process itself to the plane surfacescreen VS for display. Therefore, accurate and high values of a viewingangle may be shown. At the time of the generation of panorama image dataPD, it may not be necessary to perform difficult calculations forobtaining highly accurate horizontal and vertical viewing angles.

In step F53, in the case where it is determined not be a fullcircumference panorama image, in step F57, the display image generationsection 20 (image generation processing section 20 c) performs a displayimage data generation process of another system. For example, displayimage data PDdsp is generated as a scroll display or a single viewdisplay. Then, in step F58, a process is performed which outputs displayimage data PDdsp. The processes of these steps F57 and F58 are continuedup to it is determined to be a display end, in step F59. In this way, inthe display section, a scroll display or a single view display otherthan a projection display is executed.

In the case where it is not a full circumference panorama image, aprojection display will not be able to be executed with a high quality,without the viewing angle being exactly known. Accordingly, a scrolldisplay or a single view display is performed.

In the first configuration example such as described above, in the imageprocessing apparatus 1, the association section 11 generates a panoramaimage file PDr by associating metadata MT including information whichshows that it is a full circumference panorama image with the panoramaimage data PD.

Combination projection surface information which shows the type ofprojection surface of the frame image data at the time of panorama imagegeneration, 360° viewing angle determination information of the panoramaimage data PD, both end alignment information, panorama mode informationand horizontal and vertical viewing angle information are included inthe metadata MT.

On the other hand, in the image processing apparatus 2, the displayimage generation section 20 generates display image data by performing adisplay projection process at the time when the panorama image data PDis determined to be a full circumference panorama image, and causes aprojection display to be executed. The determination of a fullcircumference panorama image uses information which is associated as themetadata MD.

By such a configuration, a projection display can be automaticallyexecuted for a full circumference panorama image in which an accurateprojection display can be executed, and a display with a sense ofimmersion can be provided for a user. On the other hand, by performing adisplay by another display system, at the time when a viewing anglewhich is not a full circumference panorama image is not able to beaccurately specified, switching of a display operation can be performedso that a low-quality projection display is not performed.

Further, panorama mode type information, 360° viewing angledetermination information, both end alignment information or the likeare included as the metadata MT, and by having these associated with thepanorama image data PD, a determination of whether or not it is a fullcircumference panorama image can be easily and accurately executed. Thatis, the selection of a display system becomes easy and accurate.

Further, by having combination projection surface information includedin the metadata MT, a display projection process can be accuratelyperformed.

4. Second Configuration Example (Dedicated Program)

A second configuration example of an embodiment will be described. Inthe second configuration example, the association section 11 of theimage processing apparatus 1 associates a display processing program (adedicated program) used for the display of a full circumference panoramaimage with the panorama image data PD, as information which shows thatit is a full circumference panorama image.

Further, the display image generation section 20 of the image processingapparatus 2 determines that it is a full circumference panorama image,by having the display processing program associated with the panoramaimage data PD, and generates display image data by performing a displayprojection process based on this display processing program.

FIG. 13A shows the panorama combination section 10 and the associationsection 11 in the image processing apparatus 1. In this secondconfiguration example, the association section 11 has a displayprocessing program generation section 11 d and a data integrationsection 11 e.

The panorama combination section 10 generates panorama image data PD andoutputs panorama processing information Ip.

The panorama processing information Ip is supplied to the displayprocessing program generation section 11 d. The display processingprogram generation section 11 d causes a display processing program PGto be generated based on the panorama processing information Ip. Bydetermining whether or not the panorama image data PD is a fullcircumference panorama image, by any of panorama mode type information,360° viewing angle determination information and both end alignmentinformation in the panorama processing information Ip, the displayprocessing program generation section 11 d causes a display processingprogram PG to be generated, in the case of a full circumference panoramaimage. For example, since it is described in Flash (Adobe), QuickTimeVR(Apple) or HTML5, as the display processing program PG, reproduction anddisplay is possible by a wide range of display devices.

The data integration section 11 e generates a panorama image file PDr byintegrating the panorama image data PD generated by the panoramacombination section 10 with the display processing program PG generatedby the display processing program generation section 11 d. For example,the panorama image file PDr becomes a structure such as that of FIG.13B. That is, it is a data file structure which includes a header,panorama image data PD and display processing program PG.

In each of the apparatuses DVC1 to DVC3, DVC4 and DVC6 illustrated inFIG. 9 and FIG. 10, such a panorama image file PDr is recorded to arecording medium, or is transmitted and output. Note that, asillustrated as the panorama image file PDr, while the panorama imagedata PD may be set to a normal panorama holding form, and may add adisplay processing program PG, it may be set to a form which embeds thepanorama image data PD in a display processing program itself.

FIG. 13C shows the display image generation section 20 in the imageprocessing apparatus 2. In this second configuration example, thedisplay image generation section 20 has a data separation section 20 f,and an image generation processing section 20 c.

A panorama image file PDr, which is reproduced from a recording mediumor is received, is input to the data separation section 20 f. The dataseparation section 20 a separates the panorama image data PD and thedisplay processing program PG from this panorama image file PDr, andsupplies them to the image generation processing section 20 c.

Note that, in the case where the panorama image data PD is not a fullcircumference panorama image, the display processing program PG is notincluded in the panorama image file PDr, and therefore the dataseparation section 20 f supplies only the panorama image data PD to theimage generation processing section 20 c.

The image generation processing section 20 c generates and outputsdisplay image data PDdsp based on the panorama image data PD. At thistime, if the display processing program PG is present, display imagedata PDdsp generation is performed for a projection display, byperforming a process based on the display processing program PG. In thecase where the panorama image data PD is not a full circumferencepanorama image, and the display processing program PG is not extracted,a process for another display, for example, a scroll display or a singleview display, is performed.

Then, the generated display image data PDdsp is supplied to a displaysection which is not illustrated, or a device having a display section,and a display is executed.

In each of the apparatuses DVC1 to DVC3, DVC5 and DVC 7 illustrated inFIG. 9 and FIG. 10, this display image data PDdsp is supplied to thedisplay section 21, and a panorama image display is performed.

Process examples of the above described image processing apparatus 1 andthe image processing apparatus 2 are shown in FIG. 14.

FIG. 14A shows the processes of the panorama combination section 10 andthe association section 11 as the image processing apparatus 1.

In step F11, the panorama combination section 10 obtains an input imagegroup FMS.

In step F12, panorama image data PD is generated, by having the panoramacombination section 10 perform a panorama combination process. Thepanorama combination section 10 supplies panorama processing informationIp obtained in this panorama combination processing process to theassociation section 11 (the display processing program generationsection 11 d).

In step F13, the association section 11 (the display processing programgeneration section 11 d) confirms the panorama processing informationIp, and performs a determination of whether or not the generationpanorama image data PD is a full circumference panorama image.

Similar to that stated in the first configuration example, thedetermination of whether or not it is a full circumference panoramaimage can be considered by a system which refers to 360° viewing angledetermination information in the panorama processing information Ip, asystem which refers to both end alignment information, a system whichrefers to a panorama mode type information or the like.

In the case where it is a full circumference panorama image, it proceedsfrom step F14 to F15, and the association section 11 (the displayprocessing program generation section 11 d) causes a display processingprogram PG to be generated. In step F16, in the case where a displayprocessing program PG is generated, the association section 11 (the dataintegration section 11 b) integrates the panorama image data PD and thedisplay processing program PG, and generates a panorama image file PDrsuch as that of FIG. 13B, for example. In the case where a displayprocessing program PG is not generated, that is, in the case where it isnot a full circumference panorama image, a panorama image file PDr whichincludes panorama image data PD is generated.

In step F17, the panorama image file PDr is recorded to a recordingmedium by a recording and reproduction section which is not illustrated,or is transmitted and output to an external device, a network or thelike by a communication section which is not illustrated.

In FIG. 14B, the processes of the display image generation section 20 ofthe image processing apparatus 2 are shown.

In step F60, the display image generation section 20 (the dataseparation section 20 f) obtains the panorama image file PDr, which isreproduced in the recording reproduction section which is notillustrated, or is received in the communication section which is notillustrated.

In step F61, the display image generation section 20 (the dataseparation section 20 f) performs a separation process of the panoramaimage file PDr, and confirms whether or not a display processing programPG is included.

In step F62, the display image generation section 20 branches theprocesses in accordance with the presence or not of a display processingprogram PG.

In the case where a display processing program PG is present, in stepF63, the display processing program PG is obtained, and is set to astate in which it is used as a program for display image generation bythe image generation 20 c.

Then, the display image generation section 20 (the image generationprocessing section 20 c) proceeds to step F64, display image data PDdspis generated by performing a display projection process in accordancewith the display processing program PG, and in step F65, a process isperformed which outputs this display image data PDdsp. The processes ofthese steps F64 and F65 are continued up to it becomes a display end, instep F66, by a user operation or an automatic process. In this way, inthe display section, for example, a display so as to look over the fullcircumference, a display of an arbitrary direction or the like isexecuted as a projection display.

Note that, information of a projection surface of the panorama imagedata PD, in which the panorama image data PD has a 360° viewing angle,may be described in the display processing program PG, such ashorizontal and vertical viewing angle information for additional scrollrestriction.

In the case it is determined, in step F62, that a display processingprogram PG is not present, the display image generation section 20 (theimage generation processing section 20 c) performs, in step F67, adisplay image data generation process based on an existing displayprocessing program. For example, when display processing programs arestored as a scroll display or a single view display in the imagegeneration processing section 20 c, display image data PDdsp isgenerated as a scroll display or a single view display using these.Then, in step F68, a process is performed which outputs the displayimage data PDdsp. The processes of steps F67 and F68 are continued up toit is determined to be a display end, in step F69. In this way, in thedisplay section, a scroll display or a single view display other than aprojection display is executed. That is, a scroll display or a singleview display is performed in the case where it is not a fullcircumference panorama image.

In the second configuration example such as described above, in theimage processing apparatus 1, the association section 11 generates apanorama image file PDr by associating the display processing program PGfor executing a projection display corresponding to a full circumferencepanorama image with the panorama image data PD.

On the other hand, in the image processing apparatus 2, if a displayprocessing program PG is present, the display image generation section20 generates display image data PDdsp for a projection display byperforming a display projection process based on this display processingprogram PG.

By such a configuration, a projection display is automatically executedfor a full circumference panorama image in which an appropriateprojection display can be executed, and a display with a sense ofimmersion can be provided for a user. On the other hand, by performing adisplay by another display system, at the time when a viewing anglewhich is not a full circumference panorama image is not able to beaccurately specified, switching of a display operation can be performedso that a low-quality projection display is not performed.

5. Third Configuration Example (File Link Information)

A third configuration example of an embodiment will be described. In thethird configuration example, the association section 11 of the imageprocessing apparatus 1 performs a process which associates a data fileincluding information which shows that it is a full circumferencepanorama image with a data file including panorama image data PD.

Further, the display image generation section 20 of the image processingapparatus 2 confirms the data file associated from the data fileincluding the panorama image data PD, and determines whether or not itis a full circumference panorama image by acquiring information. Then,display image data is generated by performing a display projectionprocess in the case where it is a full circumference panorama image.

FIG. 15A shows the panorama combination section 10 and the associationsection 11 in the image processing apparatus 1. In this third embodimentexample, the association section 11 has a metadata generation section 11a and an association information addition section 11 c.

The panorama combination section 10 generates panorama image data PD,and outputs panorama processing information Ip. The panorama processinginformation Ip (panorama mode type information, 360° viewing angledetermination information, both end alignment information, combinationprojection information, horizontal and vertical viewing angleinformation or the like) is supplied to the metadata generation section11 a.

The metadata generation section 11 a generates metadata MT reflectingthe panorama processing information Ip.

As shown in FIG. 15B, the association information addition section 11 cgenerates a panorama image file PDr1 which includes the panorama imagedata PD, and generates a data file PDr2 which includes the metadata MT.In this case, file link information LK1 and LK2 are added to thepanorama image file PDr1 and the data file PDr2, respectively, and aremutually associated. The file link information LK1 is information whichspecifies the data file PDr2, and the file link information LK2 isinformation which specifies the panorama image file PDr1. Note that, thefile link information which specifies the other may be added to only oneof the panorama image file PDr1 and the data file PDr2.

In each of the apparatuses DVC1 to DVC3, DVC4 and DVC6 illustrated inFIG. 9 and FIG. 10, such a panorama image file PDr1 and data file PDr2are recorded to recording mediums, or are transmitted and output. Therecording mediums to which each of the files are recorded may bedifferent recording mediums. Further, each of the files may betransmitted to a different device, a network server or the like.

FIG. 15C shows the display image generation section 20 in the imageprocessing apparatus 2. In this third configuration example, the displayimage generation section 20 has an association information extractionsection 20 d, a corresponding data acquisition section 20 e, a metadatainterpretation section 20 b, and an image generation processing section20 c.

The panorama image file PDr1, which is reproduced from the recordingmedium or is received, is input to the association informationextraction section 20 d. The association information extraction section20 d extracts the file link information LK1 from this panorama imagefile PDr1 and supplies it to the corresponding data acquisition section20 e, and supplies panorama image data PD to the image generationprocessing section 20 c.

The corresponding data acquisition section 20 e performs an access ACLKby using the file link information LK1, and acquires the associated datafile PDr2. For example, the data file PDr2 is read by accessing aprescribed recording medium. Alternatively, the data file PDr2 isreceived by accessing a prescribed external device, network server orthe like. Then, if the corresponding data acquisition section 20 eacquires the data file PDr2, metadata MT is extracted, and is suppliedto the metadata interpretation section 20 b. That is, the metadata MTassociated with the panorama image data PD of a display target of thistime is obtained.

The metadata MT is interpreted by the metadata interpretation section 20b, and is supplied to the image generation processing section 20 c aspanorama processing information IP (panorama mode type information, 360°viewing angle determination information, both end alignment information,composition projection surface information, horizontal and verticalviewing angle information).

The image generation processing section 20 c generates and outputsdisplay image data PDdsp based on the panorama image data PD. At thistime, it is determined whether or not to perform display image dataPDdsp generation for a projection display, in accordance with thecontent of the panorama processing information Ip. That is, in the casewhere the panorama image data PD of a display target is determined to bea full circumference panorama image, a process for a projection displayis performed, and in the case where it is not a full circumferencepanorama image, a process for another display, for example, a scrolldisplay or a single view display, is performed.

Then, the generated display image data PDdsp is supplied to a displaysection which is not illustrated, or a device having a display section,and a display is executed.

In each of the apparatuses DVC1 to DVC3, DVC5 and DVC7 illustrated inFIG. 9 and FIG. 10, this display image data PDdsp is supplied to thedisplay section 21, and a panorama image display is performed.

Process examples of the above described image processing apparatus 1 andthe image processing apparatus 2 are shown in FIG. 16.

FIG. 16A shows the processes of the panorama combination section 10 andthe association section 11 as the image processing apparatus 1.

In step F21, the panorama combination section 10 obtains an input imagegroup FMS.

In step F22, panorama image data PD is generated, by having the panoramacombination section 10 perform a panorama combination process. Thepanorama combination section 10 supplies panorama processing informationIp obtained in this panorama combination processing process to theassociation section 11 (the metadata generation section 11 a).

In step F23, the association section 11 (the metadata generation section11 a) generates metadata MT based on the panorama processing informationIp.

In step F24, the association section 11 (the association informationaddition section 11 c) generates a panorama image file PDr1 whichincludes the panorama image data PD and the file link information LK1,or generates a data file PDr2 which includes the metadata MT and thefile link information LK2.

In step F25, the panorama image file PDr1 and the data file PDr2 arerecorded to a recording medium by a recording reproduction section whichis not illustrated, or are transmitted and output to an external device,a network or the like by a communication section which is notillustrated.

FIG. 16B shows the processes of the display image generation section 20of the image processing apparatus 2.

In step F71, the display image generation section 20 (the associationinformation extraction section 20 d) obtains the panorama image filePDr1 reproduced by the recording medium which is not illustrated, orreceived by the communication section which is not illustrated.

In step F72, the display image generation section 20 (the associationinformation extraction section 20 d, the corresponding data acquisitionsection 20 e, the metadata interpretation section 20 b) performs accessby using the link information LK1 extracted from the panorama image filePDr1, and acquires the data file PDr2. Then, the metadata MT isextracted from the data file PDr2, and panorama processing informationIp is obtained by interpreting this.

In step F73, the display image generation section 20 (the imagegeneration processing section 20 c) branches the processes by whether ornot the panorama image data PD of a display target of this time is afull circumference panorama image with a 360° viewing angle.

As described in the first configuration example, the determination ofwhether or not the panorama image data PD is a full circumferencepanorama image is possible by any of panorama mode type information,360° viewing angle determination information and both end alignmentinformation in the panorama processing information Ip.

In step F73, in the case where it is determined to be a fullcircumference panorama image, the display image generation section 20(the image generation processing section 20 c) proceeds to step F74,display image data PDdsp is generated by performing a display projectionprocess, and in step F75, a process is performed which outputs thisdisplay image data PDdsp. The processes of these steps F74 and F75 arecontinued up to it becomes a display end, in step F76, by a useroperation or an automatic process. In this way, in the display section,for example, a display so as to look over the full circumference, adisplay of an arbitrary direction or the like is executed as aprojection display.

Note that, in this case, since the panorama image data PD is a fullcircumference panorama image, the viewing angle used in the displayprojection process may be set to 360°.

Further, by referring to combination projection surface information inthe panorama processing information Ip, the image generation processingsection 20 c can accurately perform a re-projection process to a planesurface screen VS.

In the case where it is determined, in step F73, not to be a fullcircumference panorama image, the display image generation section 20(the image generation processing section 20 c) performs, in step F77, adisplay image data generation process of another system. For example,display image data PDdsp is generated as a scroll display or a singleview display. Then, in step F78, a process is performed which outputsthe display image data PDdsp. The processes of these steps F77 and F78are continued up to it is determined to be a display end, in step F79.In this way, in the display section, a scroll display or a single viewdisplay other than a projection display is executed. That is, in thecase where the panorama image data PD is not a full circumferencepanorama image, and the viewing angle is not accurately known, a scrolldisplay or a single view display is performed.

In the third configuration example such as described above, in the imageprocessing apparatus 1, the association section 11 generates a data filePDr2 which includes metadata MT including information which shows thatit is a full circumference panorama image, and a panorama image filePDr1 which includes panorama image data PD, and associates these by thefile link information LK1 and LK2.

On the other hand, the display image generation section 20 in the imageprocessing apparatus 2 acquires the metadata MT associated with thepanorama image data PD by using file link information, and determineswhether or not the panorama image data PD is a full circumferencepanorama image from the metadata MT. Then, display image data isgenerated by performing a display projection process at the time whenthe panorama image data PD is determined to be a full circumferencepanorama image, and a projection display is executed.

By such a configuration, a projection display is automatically executedfor a full circumference panorama image in which an appropriateprojection display can be executed, and a display with a sense ofimmersion can be provided for a user. On the other hand, by performing adisplay by another display system, at the time when a viewing anglewhich is not a full circumference panorama image is not able to beaccurately specified, switching of a display operation can be performedso that a low-quality projection display is not performed.

Note that, in FIG. 15 and FIG. 16 it is described by examples usingmetadata MT, it can also be considered by using the display processingprogram PG stated in the second configuration example.

For example, a display processing program PG which is not metadata MT isincluded in the data file PDr2 shown in FIG. 15B. It is an example inwhich, if a display processing program PG is included in the associateddata file PDr2, the display image generation section 20 performs ageneration process of display image data PDdsp for a projection displayby using this display processing program PG.

Information which shows that it is a full circumference panorama imageis included as a different file with the above described panorama imagedata PD, and modes in which these are associated can be considered forvarious types. As a related technique 1, for example, a Unique MaterialIdentifier (UMID) is well known as information which refers to videomaterial in video editing data (for example, refer to JP 2004-312233A).For example, the data file PDr2 of FIG. 15B may associate metadata MTwith panorama image data PD in the panorama image file PDr1 which is adifferent file, by a mode such as UMID.

6. Fourth Configuration Example (Dedicated Format)

A fourth configuration example of an embodiment will be described. Inthe fourth embodiment, the association section 11 of the imageprocessing apparatus 1 sets panorama image data itself to data of aspecific format, as information which shows that it is a fullcircumference panorama image.

Further, the display image generation section 20 of the image processingapparatus 2 determines that it is a full circumference panorama image,by setting panorama image data to data of a specific format, andgenerates display image data by performing a display projection process.

FIG. 17A shows the panorama combination section 10 and the associationsection 11 in the image processing apparatus 1. In this fourthconfiguration example, the association section 11 has a formatprocessing section 11 f.

The panorama combination section 10 generates panorama image data PD andoutputs panorama processing information Ip.

The format processing section 11 f determines whether or not thepanorama image data PD is a full circumference panorama image, by any ofpanorama mode type information, 360° viewing angle determinationinformation and both end alignment information in the panoramaprocessing information Ip. Then, in the case of a full circumferencepanorama image, a panorama image file PDr, which includes panorama imagedata PD of a dedicated format, is generated. In the case where it is nota full circumference panorama image, a panorama image file PDr, whichincludes panorama image data PD of a general normal format, isgenerated.

Note that, for example, panorama image data PD of a normal format is animage data form usually used such as a Joint Photographic Experts Group(JPEG) format or the like. On the other hand, a dedicated format is setto a new data format exclusively for a full circumference panoramaimage.

FIG. 17B shows the display image generation section 20 in the imageprocessing apparatus 2. In this fourth configuration example, thedisplay image generation section 20 has a format determination section20 g, and an image generation processing section 20 c.

A panorama image file PDr, which is reproduced from a recording mediumor is received, is input to the format determination section 20 g. Theformat determination section 20 g determines whether panorama image dataPD included in the panorama image file PDr is a dedicated format or ageneral format. Then, the extracted panorama image data PD and formatdetermination information FM are supplied to the image generationprocessing section 20 c.

The image generation processing section 20 c generates and outputsdisplay image data PDdsp based on the panorama image data PD. At thistime, it is determined whether or not it is a full circumferencepanorama image by the format determination information FM, andgeneration of display image data PDdsp for a projection display isperformed if it is a full circumference panorama image. In the casewhere the panorama image data PD is not a full circumference panoramaimage, a process for another display, for example, a scroll display or asingle view display, is performed.

Then, the generated display image data PDdsp is supplied to a displaysection which is not illustrated, or a device having a display section,and a display is executed.

Process examples of the above described image processing apparatus 1 andthe image processing apparatus 2 are shown in FIG. 18.

FIG. 18A shows the processes of the panorama combination section 10 andthe association section 11 as the image processing apparatus 1.

In step F41, the panorama combination section 10 obtains an input imagegroup FMS.

In step F42, panorama image data PD is generated, by having the panoramacombination section 10 perform a panorama combination process. Thepanorama combination section 10 supplies panorama processing informationIp obtained in the panorama combination process processing, with thepanorama image data PD, to the association section 11 (the formatprocessing section 11 f).

In step F43, the association section 11 (the format processing section11 f) confirms the panorama processing information Ip, and performs adetermination of whether or not the generated panorama image data PD isa full circumference panorama image. Similar to that stated in the firstconfiguration example, the determination of whether or not it is a fullcircumference panorama image can be considered by a system which refersto 360° viewing angle determination information in the panoramaprocessing information Ip, a system which refers to both end alignmentinformation, a system which refers to panorama mode type information orthe like.

In the case where it is a full circumference panorama image, it proceedsfrom step F44 to F45, and the association section 11 (the formatprocessing section 11 f) generates a panorama image file PDr byconverting the panorama image data PD into a dedicated format.

In the case where it is not a full circumference panorama image, itproceeds from step F44 to F46, and the association section 11 (theformat processing section 11 f) generates a panorama image file PDrwhich includes panorama image data PD of a normal format.

In step F47, the panorama image file PDr is recorded to a recordingmedium by a recording and reproduction section which is not illustrated,or is transmitted and output to an external device, a network or thelike by a communication section which is not illustrated.

FIG. 18B shows the processes of the display image generation section 20of the image processing apparatus 2.

In step F81, the display image generation section 20 (the formatdetermination section 20 g) obtains a panorama image file PDr reproducedby a recording reproduction section which is not illustrated, orreceived by a communication section which is not illustrated.

In step F82, the display image generation section 20 (the formatdetermination section 20 g) determines the format of panorama image dataPD included in the panorama image file PDr.

Then, in step F83, the display image generation section 20 branches theprocesses in accordance with a format determination result.

In the case where it is determined to be a dedicated format, the displayimage generation section 20 (the image generation processing section 20c) proceeds to step F84, display image data PDdsp is generated byperforming a display projection, and in step F85, this display imagedata PDdsp is output. The processes of these steps F84 and F85 arecontinued up to it becomes a display end, in step F86, by a useroperation or an automatic process. In this way, in the display section,for example, a display so as to look over the full circumference, adisplay of an arbitrary direction or the like is executed as aprojection display.

Note that, the viewing angle of the panorama image data PD, which isused by a projection calculation in the display projection process, maybe set to 360°.

Further, information of a projection surface of the panorama image dataPD, horizontal and vertical viewing angle information for scrolllimitation or the like may be considered by being embedded in the dataform of a dedicated format.

In step F83, in the case where it is determined that it is not adedicated format, the display image generation section 20 (the imagegeneration processing section 20 c) generates, in step F87, displayimage data PDdsp as another system, for example, a scroll display or asingle view display. Then, in step F88, a process is performed whichoutputs the display image data PDdsp. The processes of these steps F87and F88 are continued up to it is determined to be a display end, instep F89. In this case, in the display section, a scroll display or asingle view display other than a projection display is executed.

In the fourth configuration example such as described above, in theimage processing apparatus 1, the association section 11 makes adedicated format in the case of a full circumference panorama image.

On the other hand, in the image processing apparatus 2, by determiningthat the panorama image data PD is a full circumference panorama imageif it is a dedicated format, the display image generation section 20generates display image data PDdsp for a projection display byperforming a display projection process.

By this configuration, a projection display is automatically executedfor a full circumference panorama image in which an appropriateprojection display can be performed, and a display which has a sense ofimmersion can be provided for a user. On the other hand, by performing adisplay by another display method, at the time when a viewing anglewhich is not a full circumference panorama image is not able to beaccurately specified, switching of a display operation can be performedso that a low-quality projection display is not performed.

7. Fifth Configuration Example (Application Example to an ImagingApparatus)

(7-1: Configuration of the Imaging Apparatus)

To continue, as a fifth configuration example of an embodiment, animaging apparatus 50 will be described as a specific example by a devicewhich includes the image processing apparatuses 1 and 2 of the presentdisclosure.

FIG. 19 shows a configuration example of the imaging apparatus 50.

The imaging apparatus 50 includes a lens unit 100, an imaging element101, an image processing section 102, a control section 103, a displaysection 104, a memory section 105, a recording device 106, an operationsection 107, a sensor section 108, and a communication section 109.

The lens unit 100 collects an optical image of a photographic subject.The lens unit 100 has a function which adjusts a focal distance, aphotographic subject distance, an aperture or the like, so as to obtainan appropriate image, in accordance with an instruction from the controlsection 103. Further, it also has a camera-shake correction function foroptically suppressing blurring of an image.

The imaging element 101 optically converts the optical image collectedby the lens unit 100, and converts it into electrical signals.Specifically, it is implemented by a Charge Coupled Device (CCD) imagesensor, a Complementary Metal Oxide Semiconductor (CMOS) image sensor orthe like.

The image processing section 102 is constituted from a sampling circuitwhich samples electrical signals from the imaging element 101, an A/Dconversion circuit which converts analog signals into digital signals,an image processing circuit which applies a prescribed image process todigital signals or the like. Here, this image processing section 102 isshown as performing a process which obtains frame image data by imagingwith the imaging element 101, and performing a process which combines apanorama image.

This image processing section 102 includes not only a dedicated hardwarecircuit, but also a Central Processing Unit (CPU) or a Digital SignalProcessor (DSP), and can perform software processes to accommodateflexible image processes.

The control section 103 is constituted from a CPU and a control program,and performs a control of each section of the imaging apparatus 50. Thecontrol program itself is actually stored in the memory section 105, andis executed by the CPU.

The processes of the panorama combination section 10 and the associationsection 11 disclosed in each of the above described configurationexamples, and additionally the processes of the display image generationsection 20, are executed by the control section 103 and the imageprocessing section 102. Therefore, the image processing apparatuses 1and 2 of the present disclosure are implemented by the control section103 and the image processing section 102.

The display section 104 is constituted from a D/A conversion circuitwhich makes image data processed by the image processing section 102 andstored in the memory section 105 analog, a video encoder which encodesthe image signals made analog in video signals of a form which isappropriate for a display device of a later stage, and a display devicewhich displays an image corresponding to the input video signals.

The display device is implemented, for example, by a Liquid CrystalDisplay (LCD), an organic Electroluminescence (EL) panel or the like,and also has a function as a finder.

The memory section 105 is constituted from a semiconductor memory suchas a Dynamic Random Access Memory (DRAM), and image data processed bythe image processing section 102, control programs and various types ofdata in the control section 103 or the like are temporarily recorded.

The recording device 106 is constituted by a recording medium such as asemiconductor memory such as a flash memory (Flash Memory), a magneticdisc, an optical disc or a magneto-optical disc, and a recording andreproduction circuit/mechanism for these recording mediums.

At the time of imaging by the imaging apparatus 50, encoding isperformed in a Joint Photographic Experts Group (JPEG) form by the imageprocessing section 102, and JPEG image data stored in the memory section105 is recorded to a recording media.

At the time of reproduction, JPEG image data held in a recording mediumis read to the memory section 105, and a decoding process is performedby the image processing section 102. It is possible for the decodedimage data to be displayed by the display section 104, or to betransmitted and output to an external device by the communicationsection 109.

The operation section 107 includes input devices such as hardware keyssuch as a shutter button, up, down, left and right arrow keys, adetermination key and a cancel key, an operation dial, a touch panel anda zoom lever, detects an input operation of a photographer (user), andtransmits it to a control section 103. The control section 103determines an operation of the imaging apparatus 50 in accordance withan input operation of the user, and performs a control so that eachsection performs necessary operations.

The sensor section 108 is constituted by a gyro sensor, an accelerationsensor, a geomagnetic sensor, a Global Positioning System (GPS) sensoror the like, and performs detection of various types of information.Apart from being added as metadata for captured image data, thisinformation is also used for various types of image processes andcontrol processes.

The communication section 109 performs communication with an externaldevice or network communication. Specifically, various types of examplescan be considered for the communication section 109, such as a USBcommunication section, a LAN communication section, a public networkcommunication section such as the internet, or a dedicated communicationsection with a specific device. Further, the form of a transmission linemay be either a wireless transmission line by electronic waves orinfrared rays, or a cable transmission line by a cable connection. Thesignal mode can be assumed to be digital electronic signalcommunication, analog electronic signal communication, opticalcommunication or the like.

The image processing section 102, the control section 103, the displaysection 104, the memory section 105, the recording device 106, theoperation section 107, the sensor section 108 and the communicationsection 109 are mutually connected via a bus 110, and image data,control signals or the like are exchanged.

(7-2: Panorama Combination and Association Processes)

Next, a panorama combination process and an association process of theimaging apparatus 50 of the present embodiment will be described indetail.

FIG. 20 shows, as functional configurations, the processes executed inthe image processing section 102 and the control section 103 for thepanorama combination process and the association process, and shows theprocesses executed by these functional configuration portions.

Note that, here, the example using metadata stated in the abovedescribed first configuration example will be described as theassociation process.

As shown by the dashed line in FIG. 20, the panorama combination section10 and the association section 11 are included as functionalconfigurations. That is, the image processing apparatus 1 is formedinside the imaging apparatus 50, by having a function included as theassociation section 11 by at least the image processing section 102, thecontrol section 103 or the like.

Further, the process functions executed by the functions as the panoramacombination section 10 will be described. In FIG. 20, the processes(algorithm flow) executed as the panorama combination section 10 areshown by each block.

That is, in the panorama combination section 10, a pre-process 200, animage registration process 201, a moving photographic subject detectionprocess 202, a detection/recognition process 203, a 360° imagingdetermination process 204, a 360° optimization process 205, acombination projection process 206, a seam determination process 207 anda stitch process 208 are performed.

Each of the processes will be described.

As described in FIG. 1, an input image group FMS which becomes a targetof the pre-process 200 is frame image data FM #0, FM #1, FM #2 . . .sequentially obtained at the time when a photographer executes panoramaimaging by the imaging apparatus 50.

In the panorama combination section 10, first, the pre-process 200 forpanorama combination is performed for images (each frame image data)captured by a panorama imaging operation of a photographer. Note that,here, an image process the same as that at the time of normal imaging isperformed for the frame image data FM.

The input image is influenced by aberrations based on thecharacteristics of the lens unit 100. In particular, distortionaberrations of the lens will negatively affect the image registrationprocess 201, and will reduce the accuracy of alignment. In addition,since artifacts are also generated in the seam vicinity of a combinedpanorama image, correction of the distortion aberrations is performed inthis pre-process 200. By the correction of the distortion aberrations,there will be an effect in which the accuracy of the moving photographicsubject detection process 202 and the detection/recognition process 203are improved.

Next, in the panorama combination section 10, the image registrationprocess 201, the moving photographic subject detection process 202 andthe detection/recognition process 203 are performed for the frame imagedata to which the pre-process 200 has been performed.

It may be necessary to convert a plurality of frame image data intocoordinates in a simple coordinate system at the time of panoramacombination, and this simple coordinate system will be called a panoramacoordinate system.

The image registration process 201 is a process which inputs twosuccessive frame image data, and performs alignment in the panoramacoordinate system. While information obtained by the image registrationprocess 201 for two frame image data is a relative relation between thetwo image coordinates, a coordinate system of all the frame image datacan be converted into a panorama coordinate system, by selecting one outof the plurality of image coordinate systems (for example, thecoordinate system of the initial frame image data), and fixing it to thepanorama coordinate system.

The specific processes performed in the image registration process 201will be roughly divided into two as follows.

1. Detecting local movements within an image.

2. Obtaining full image global movements from the above describedobtained local movement information.

In the above described process 1, generally:

-   -   Block matching    -   Characteristic point extraction and characteristic point        matching such as Harris, Hessian, SIFT, SURF or FAST

or the like are used, and local vectors of characteristic points of theimages are obtained.

In the above described process 2, robust estimation techniques such as:

-   -   A Least Squares Method    -   An M-Estimator    -   A Least Median Squares (LMedS) method    -   A RANdom SAmple Consensus (RANSAC)

is used as a local vector group obtained in the above described process1, and an optimal affine transformation matrix or a projectiontransformation matrix (Homography), which states a relation between 2coordinate systems, is obtained. In the present disclosure, thisinformation will be called image registration information.

Further, in the panorama composition process, a part of a movingphotographic subject in which a moving photographic subject is presentin an imaging scene is divided in terms of the characteristics in whicha plurality of frame image data is combined, and will cause a breakdownof an image such as becoming blurry or a reduction of the image quality.Accordingly, it is preferable to determine the seam of a panorama, upondetecting a moving photographic subject, by avoiding the movingphotographic subject.

The moving photographic subject detection process 202 is a process whichinputs 2 or more successive frame image data, and performs detection ofa moving photographic subject. In the case where an example of aspecific process is at a threshold or above where there is a differencein values of the pixels of the 2 frame image data in which alignment isactually performed, by image registration information obtained by theimage registration process 201, these pixels will be determined to be amoving photographic subject.

Alternatively, a determination may be performed by using characteristicpoint information, which is determined as outliers (outliers) at thetime of robust estimation of the image registration process 201.

In the detection/recognition process 203, position information such as ahuman face or body or an animal within captured frame image data isdetected. Even if there is a high possibility of a human or animal beinga photographic subject, which is not moving, in the case where the seamof the panorama is determined on this photographic subject, a sense ofincompatibility will often occur visually compared to other objects, andtherefore it is preferable to determine a seam by avoiding theseobjects. That is, information obtained in this detection/recognitionprocess 203 is used for supplementing information of the movingphotographic subject detection process 202.

A seam (junction) is determined in the seam determination process 207,which will be described later, based on information obtained in themoving photographic subject detection process 202 and thedetection/recognition process 203.

The 360° imaging determination process 204 and the 360° optimizationprocess 205 perform processes specialized for 360° full circumferenceimaging.

Here, in order to understand these processes, various problems at thetime of 360° imaging will be described.

FIG. 21A, FIG. 21B and FIG. 22 are examples of panorama images in eachof the cases. Note that, in these figures, in order to confirmcontinuality of the left end and the right end of the panorama images, apart of the left end regions of the panorama images are shown copied(image RCP) onto each of the right ends of the panorama images (theright side of the boundary line BD). An image portion in which the imageRCP is excluded, that is, a portion shown by the range RP of the outputpanorama image, is a final one panorama image.

In the case where it is a 360° full circumference panorama, it isdesirable for an image having continuality without gaps to be naturallyconnected on both sides of this boundary line BD.

In a panorama image in which a plurality of still images are combined,it will be difficult to accurately know a general panorama viewingangle. Usually, a focal distance (f: pixel conversion value) of the lensof the imaging apparatus, a pixel number (1) of a sweep direction of thepanorama image, and a panorama viewing angle (θ) have the followingrelation.1=fθ

However, since there are actually influences such as:

-   -   A radius of gyration when sweeping    -   Correction errors of distortion aberrations    -   Errors of a registration process, there can be significant        differences with the viewing angle of an actual scene.

In the case where the registration process is limited to only parallelmovement, and simplification is performed on an algorithm such asomitting a projection process, for the processing performance of theimaging apparatus, these errors will become more pronounced.

FIG. 21A is an example of a panorama image, to which a suitable 360°panorama image number is determined from the above equation, and acomposition process is performed. It is combined by significantlyexceeding 360° for errors, and there is no continuality of the left endand the right end (boundary line BD portion) of the panorama image.

In this way, while it is difficult to accurately determine a generalpanorama viewing angle, in the case where it is a 360° fullcircumference image, it is possible to determine a viewing angle byconsidering a constraint condition in which continuity of the left endand the right end of a panorama is obtained.

FIG. 23 will be referred to. FIG. 23 is a schematic diagram of an imagegroup (frame image data FM #0 to FM #(n−1)) obtained by 360° panoramaimaging.

Specifically, the following conditions are established, at the time whenthe sweep direction is assumed to be from left to right, and the firstframe image data FM #0 used on the panorama left side is set to a 0°position, for the frame image data FM #0A of an image entirely the sameas this, which is at a position prior to 360°, and the final frame imagedata FM #(n−1) used on the right end of the panorama image.

-   -   FM #0A and FM #(n−1) have overlapping regions.    -   FM #(n−1) is an image nearer to FM #0A on the left side than FM        #0A.

By retrieving the frame image data FM #(n−1) as a final image whichsatisfies these conditions, a 360° position can be fixed.

In the imaging apparatus 50 of this example, this process is performedby the 360° imaging determination process 204.

In the 360° imaging determination process 204, in the case where thefinal frame image data FM #(n−1) which satisfies the above described twoconditions can be determined, it is determined that the panorama imagedata within the combination process is a full circumference panoramaimage by 360° imaging. On the other hand, in the case where the finalframe image data FM #(n−1) which satisfies the above described twoconditions is not able to be determined, it is determined that thepanorama image data within the combination process is not a fullcircumference panorama image.

This determination result becomes 360° viewing angle determinationinformation which is one of the above described panorama processinginformation Ip. In the 360° imaging determination process 204, a processis performed which outputs the 360° viewing angle determinationinformation to the association section 11.

Note that, this 360° imaging determination process 204 may be performedin the case where panorama imaging is executed in a 360° panoramaimaging mode. In the cases other than a 360° panorama imaging mode (forexample, a 180° panorama imaging mode or the like), it may not benecessary to performed this 360° imaging determination process 204. Notethat, in this case, it may be set to information which shows it is anon-full circumference panorama image as the 360° viewing angledetermination information.

Further, in the process of this 360° imaging determination process 204,it may be output to the association section 11 by generating horizontaland vertical viewing angle information and panorama mode typeinformation.

FIG. 21B is an example of a result of combining the final imagedetermined in the 360° imaging determination process 204, that is, up tothe frame image data FM #(n−1) which conforms to the above described twoconditions.

Since a 360° position is determined in the 360° imaging determinationprocess 204, it will become a 360° image as a panorama image. However,this example does not have continuality of the left end and the rightend of a panorama similar to that of the example of FIG. 21A. While theposition of the horizontal direction is suitable in this example, theposition of the vertical direction will shift.

Since this performs alignment of each image between adjacent images, itwill be generated by having errors accumulated in the registrationprocess.

That is, in FIG. 23, there is a phenomenon in which a position (aposition shown as the frame image data FM #0B) to be a first image priorto 360° for becoming a full circumference panorama with continuality iscalculated for a position shown as the frame image data FM #0A for anaccumulated error of the registration process.

In order to address this phenomenon, in the following 360° optimizationprocess 205, in the case where it is determined to be 360° fullcircumference imaging in the 360° imaging determination process 204, theimage registration information is optimized, by using the imageregistration information input from the image registration process 201,and the final image information determined by the 360° imagingdetermination process 204, and is appropriately corrected as a 360° fullcircumference panorama.

This process is the previously stated process of both end alignment.That is, image registration information is adjusted so that there are noimage gaps in the vertical direction and the horizontal direction, atboth ends of the panorama image which are the 0° and 360° positions ofthe panorama image.

When described in FIG. 23, the positions of all the frame image data FM#0 to FM #(n−1) become optimized and corrected, so that the position ofthe frame image data FM #0A matches the position of the frame image dataFM #0B.

Specifically, there is a technique which simply distributes accumulatederrors to all the image registration information, a technique whichperforms optimization without impairment as long as the originalregistration information can be made using a least squares method or thelike.

For example, a technique which simply distributes is that which isstated as follows. There are errors ΔX of the horizontal direction anderrors ΔY of the vertical direction such as shown in FIG. 23 at theposition of the frame image data FM #0A and the position of the frameimage data FM #0B. The x-coordinate positions and the y-coordinatepositions of n frame image data FM #0 to FM #(n−1) constituting thepanorama image are each respectively shifted ΔX/n and ΔY/n. In this way,errors will be absorbed little by little in each frame image data FM,and the positions of virtual frame image data FM #0A as an image thesame as that of the first frame image data FM #0 will match thepositions shown as the frame image data FM #0B.

While each of the frame image data FM #0 to FM #(n−1) will shiftslightly more than the original coordinates, the entire image qualitywill have hardly any deterioration.

FIG. 22 is an example of a result in which panorama combination isperformed by using image registration information corrected by having aboth end alignment performed such as described above in the 360°optimization process 205. By the 360° imaging determination process 204and the 360° optimization process 205, it can be confirmed that an imagewill be naturally connected, without vertical and horizontal gaps atboth ends of a panorama, as the range RP of an output panorama image.

By the 360° optimization process 205 such as described above,information of whether or not a both end alignment is executed becomesboth end alignment information which is one of the panorama processinginformation Ip.

In the 360° optimization process 205, a process is also performed whichoutputs the both end alignment information to the association section11.

Next, in the combination projection process 204, a projection process isperformed on a single plane surface or a single curved surface such as acylindrical surface or spherical surface for all the frame image data,based on the image registration information obtained by the imageregistration process 201 (or image registration information corrected inthe 360° optimization process 205). Further, at the same time, aprojection process is performed on the same plane surface or curvedsurface for moving photographic subject information anddetection/confirmation information. A projection surface may beautomatically selected from the viewing angle or the like at the time ofimaging, or may be set in accordance with a user operation.

By considering optimization of pixel processing, the combinationprojection process 204 of frame image data may be performed as a processpreceding the stitch process 208, or as a part of the stitch process208. Further, for example, it may be simply performed as a part of thepre-process 200, prior to the image registration process 201. Further,by being additionally simplified, it may be handled as an approximationof a cylindrical projection process without performing this processing.

Note that, the type of information of a projection surface in thiscombination projection process 206 becomes combination projectionsurface information which is one of the panorama processing informationIp.

In the combination projection process 206, a process is also performedwhich outputs the combination projection surface information to theassociation section 11.

The seam determination process 207 is a process which sets, to an input,image data from the combination projection process 206, imageregistration information from the image registration process 201, movingphotographic subject information from the moving photographic subjectdetection process 202 and detection/confirmation information from thedetection/recognition process 203, and determinates appropriate seams SM(seams SM0 to SM(n−2) described in FIG. 2 and FIG. 3), which have littlefailure as a panorama image.

In the seam determination process 207, first, a cost function is definedfor overlapping regions between adjacent frame image data from the inputinformation. For example, a total value may be set to a function value,which associates each appropriate overlap with the moving photographicsubject information from the moving photographic subject detectionprocess 202 and detection/confirmation information from thedetection/recognition process 203, for each pixel of the overlappingregions.

In this case, since it has the meaning of the cost function valueincreasing with an increase in objects such as moving photographicsubjects being present at this point, a collection of points with lowcost function values may be seamed, in order for failures in thepanorama image to be suppressed to a minimum.

In the case where there are n images (frame image data) used forpanorama image combination, the number of these overlapping regions willbecome n−1, and the cost functions will be defined as n−1. Therefore, inorder for optimal seams to be selected as an entire panorama image, acombination is obtained in which these n−1 cost functions are minimized.This is generally called a combination optimization problem, and thefollowing solutions are well-known.

-   -   Methods for obtaining an exact solution    -   Branch and bound method    -   Memoization    -   Dynamic Programming technique (Dynamic Programming)    -   Graph cut    -   Methods which obtain an approximate solution    -   Local search method (Hill-climbing method)    -   Annealing method (Simulated Annealing)    -   Taboo search    -   Generic algorithm (Generic algorithm)

All of the seams SM1 to SM(n−2) can be obtained by any of the abovedescribed methods.

In the stitch process 208, a final panorama image is combined, by usingall of the seams SM1 to SM(n−2) determined such as described above, andeach of the frame image data FM #0 to FM #(n−1).

A blend process for reducing unnaturalness of junctions is performed forthe regions of the seam surroundings, copying of simple pixels, that is,re-sampling to a panorama coordinate system, is only performed for theregions other than these, and the entire image is connected.

By finally sweeping by considering a camera shake amount and trimmingunnecessary portions of the vertical direction, a panorama image(panorama image data PD) with a wide viewing angle can be obtained, suchas that which has a sweep direction set to a long side direction.

The above become the processes of the panorama combination section 10.

In the case of the present embodiment, the processes of the associationsection 11 are additionally performed for the panorama image data PDgenerated by the panorama combination section 10.

In the functional configuration example of FIG. 20 in the imagingapparatus 50 of this example, the previous functional configuration ofFIG. 11A is adopted as the first configuration example. The associationsection 11 performs a metadata generation process 301 and a dataintegration process 302.

Various types of panorama processing information Ip are supplied fromthe panorama combination section 10 to the association section 11. Thatis, 360° viewing angle determination information obtained in the 360°imaging determination process 204, both end alignment informationobtained in the 360° optimization process 205, and combinationprojection surface information obtained in the combination projectionprocess 206 are supplied to the association section 11.

Note that, panorama mode type information and horizontal and verticalviewing angle information may also be additionally supplied.

The association section 11 generates metadata MT reflecting the panoramaprocessing information Ip as a metadata generation process 301.

Note that, since a determination of whether or not it is a fullcircumference panorama image is performed in the processes of thedisplay image generation section 20, which will be at least describedlater, and from the viewpoint of being used for performing a displayprojection process, it may not be necessary for the metadata MT toinclude all of the 360° viewing angle determination information, bothend alignment information, combination projection surface information,panorama mode type information and horizontal and vertical viewing angleinformation.

While it has been described in the first configuration example to thefourth configuration example, the determination of whether or not it isa full circumference panorama image by the display image generationsection 20 may refer to any of the 360° viewing angle determinationinformation, both end alignment information and panorama mode typeinformation. However, combination projection surface information may benecessary for appropriately performing a display projection process.

Accordingly, at least one of:

-   -   360° viewing angle information and combination projection        surface information    -   Both end alignment information and combination projection        surface information    -   Panorama mode type information and combination projection        surface information

may be included as the panorama processing information Ip, generated bythe panorama combination section 10, and made into metadata by themetadata generation section 11 a.

In the data integration process 302, the association section 11generates a panorama image file PDr by integrating the panorama imagedata PD generated by the panorama combination section 10 with themetadata MT generated by the metadata generation process 301.

For example, the panorama image file PDr becomes a structure which hasimage data and metadata such as that disclosed in FIG. 11B. Morespecifically, an example can be considered in which metadata such as anExchangeable Image File Format (EXIF) is embedded within the image data.Note that, it may be set to a file in which the panorama image data PDand the metadata MT are independent.

In the imaging apparatus 50 of the present example, processes areperformed as the above described panorama combination section 10 andassociation section 11 in the image processing section 102 and thecontrol section 103, and as a result of this, the panorama image filePDr is recorded to the recording device 106.

Note that, the panorama image file PDr may be externally transmittedfrom the communication section 109.

An example of the series of processes from panorama imaging up torecording, which includes the processes of the above described panoramacombination section 10 and association section 11, is shown in FIG. 24.

The image capturing of step F100 captures one still image in a panoramaimaging mode, and means an obtaining process as one frame image datawithin the imaging apparatus 1. That is, imaging signals obtained by theimaging element section 101 are imaging signal processed by the imageprocessing section 102, by a control of the control section 102, andbecome one frame image data.

This frame image data may be supplied to the panorama combinationprocesses (the processes of the panorama combination section 10 of FIG.20) by the image processing section 102 as it is, or it may be suppliedto the panorama combination processes by the image processing section102 as one frame image data after being obtained once in the memorysection 105.

The processes from step F101 onwards are performed, in accordance withan input of frame image data based on step F100, by the panoramacombination section 10 of FIG. 20 implemented by the image processingsection 102 and the control section 103.

In step F101, the panorama combination section 10 performs thepre-process 200.

In step F102, the panorama combination section 10 performs the imageregistration process 201.

In step F103, the panorama combination section 10 performs the movingphotographic subject detection process 202.

In step F104, the panorama combination section 10 performs thedetection/recognition process 203.

Note that, pixel information, image registration information, movingphotographic subject information, detection/confirmation information orthe like of an image obtained in each of these processes are temporarilystored in the memory section 105 as information corresponding to frameimage data of respective process targets.

The above described processes are performed for each frame image data,up to it is an imaging end, in step F105.

If the above described processes are finished for all the obtained frameimage data FM, in step F106, the panorama combination section 10performs the 360° imaging determination process 204.

In step F107, the panorama combination section 10 performs the 360°optimization process 205.

Then, in step F108, the panorama combination section 10 performs thecombination projection process 206, by using image registrationinformation or the like adjusted in the 360° optimization process 205.

In step F109, the panorama combination section 10 determines the seamsSM0 to SM(n−2), by performing the seam determination process 207.

In step F110, the panorama combination section 10 executes the stitchprocess 208. That is, each frame image data is connected by each of theseams SM0 to SM(n−2). A blend process is also performed when beingconnected. Hereinafter, one panorama image data PD is generated such asthat shown in FIG. 3A.

To continue, the association section 11 performs the metadata generationprocess 301. That is, metadata MT is generated which reflects thepanorama processing information IP supplied from the panoramacombination section 10.

Then, in step F112, the association section 11 generates a panoramaimage file PDr by performing the data integration process 302.

The control section 103 performs a control which causes the panoramaimage file PDr generated in such a process as the panorama combinationsection 10 and association section 11 to be recorded to the recordingdevice 106.

In the above described processes, the panorama image file PDr whichincludes the panorama image data PD and the metadata MT is recorded, andreproduction and display or the like becomes possible after this.

(7-3: Panorama Image Display)

To continue, an operation will be described in the case where therecorded panorama image file PDr such as described above is selected anddisplayed in the imaging apparatus 50.

First, the display operation modes performed by the display section 104of the imaging apparatus 50 will be described in FIG. 25.

In the imaging apparatus 50, display operations of the above describedfour display modes of a single view display, a list display, a scrolldisplay and a projection display are capable of being executed. FIG. 25shows state transitions of the four display modes.

In order for simplification of the description, only a plurality ofpanorama image files PDr will be recorded to the recording device 106.

First, when a user specifies reproduction and display by using keys orthe like of the operation section 107, the control section 103 sets theimaging apparatus 50 to a reproduction and display state. At this time,it transitions from an initial state to a single view display mode.Here, for example, the latest image of the panorama image file PDr isdisplayed such as that of FIG. 7A, for example. In this single viewdisplay mode, for example, it can be switched other images of thepanorama image file PDr by having the user press the right arrow key andthe left arrow key of the operation section 107.

Further, in a single view display, the state transitions to a listdisplay such as that of FIG. 7B when the upper arrow key is pressed, anda list view of images is performed.

The state transitions again to a single view display, by having the userselect an image to be a target using the up, down, left and right arrowkeys, on this list display screen, and press the determination key, andthe selected image of the panorama image file PDr is displayed.

In the case where the determination key is pressed, for example, in asingle view display, an enlarged display is performed. Note that, here,an enlarged display will be a scroll display or a projection display.

The control section 103 refers to the metadata of the displayed image ofthe panorama image file PDr, and in the case where the panorama imagefile PDr is a full circumference panorama image, it transitions thestate to a projection display mode, and a projection display isexecuted. In this case, a projection process is performed, in accordancewith projection surface information at the time of combining which isrecorded as metadata, and a projection display is started.

In this projection display mode, it is possible for a user to specify achange of viewing direction by using the up, down, left and right arrowkeys, or a zoom-in, zoom-out by using the zoom lever. Further, in thiscase, by having horizontal and vertical viewing angle informationrecorded in metadata for a viewing direction and zoom value, alimitation is performed by the horizontal viewing angle or the verticalviewing angle as a panorama image, and therefore will not refer to thatoutside of the image.

In the case where there is no key operation for a fixed time, it ispossible for an automatic viewing change to be performed by a menusetting.

In this projection display mode, it transitions to a single view displaystate when the user presses a cancel key, and a single view display isperformed again.

In the case where the determination key is pressed in a single viewdisplay, and the displayed image of the panorama image file PDr is not afull circumference panorama image, it transitions to a scroll displaymode, and a scroll display is executed.

In a scroll display mode, while scrolling is automatically started inthe long side direction of a panorama, it is possible for the user toalso specify scrolling in an arbitrary direction by using the up, down,left and right arrow keys, or a zoom-in or zoom-out by using the zoomlever.

In a scroll display mode, it transitions to a single view display statewhen the user presses a cancel key, and a single view display isperformed again.

In the imaging apparatus 50 of the present embodiment with such displaymode transitions as described above, a projection display is performedif it is a full circumference panorama image, and automatic switching ofdisplay operations, such as a scroll display being performed if it isnot a full circumference panorama image, is performed.

If it is a full circumference panorama image, a high-quality projectiondisplay is automatically provided for the user. On the other hand, inthe case where it is not a full circumference panorama image, it ispossible for image distortions or the like to occur when performing aprojection display, and a high-quality display is not be able to beguaranteed. Accordingly, a display of a panorama image is performed in ascroll display. That is, an appropriate display mode is selected inaccordance with the panorama image.

Configurations and processes for performing such displays will bedescribed in FIG. 26 and FIG. 27.

FIG. 26 shows the processes executed in the display image generationsection 20 included in the control section 103 for the generationprocess of display image data PDdsp.

Note that, an example based on the example using metadata stated in theabove described first configuration example will be set as the processesof the display image generation section 20.

The display image generation section 20 executed by software in thecontrol section 103 performs a data separation process 401, a metadatainterpretation process 402 and an image generation process 403 such asillustrated.

The display image generation section 20 separates metadata MT andpanorama image data PD, as the data separation process 401, for apanorama image file PDr of a display target read from the recordingdevice 106.

The separated metadata MT is interpreted by the metadata interpretationprocess 402, and is supplied to the image generation process 403 aspanorama processing information Ip (panorama mode type information, 360°viewing angle determination information, both end alignment information,combination projection surface information, horizontal and verticalviewing angle information).

In the image generation process 403, display image data PDdsp isgenerated in accordance with display mode information Md, and is outputto the display section 104.

A display system selection process 403 a, a single view display process403 b, a list display process 403 c, a scroll display process 403 d anda projection display process 403 e are performed as the image generationprocess 403.

In the display system selection process 403 a, a process is performedwhich selects what display is to be executed, based on the display modeinformation Md corresponding to the above described display modetransitions based on a user operation or the like. Further, when movingto a scroll display mode or a projection display mode at the time of astate of a single view display mode, a scroll display mode and aprojection display mode are selected, by referring to the panoramaprocessing information Ip interpreted in the metadata interpretationprocess 402.

In the single view display process 403 b, display image data PDdsp isgenerated as a single view display for the supplied panorama image dataPD.

In the list display process 403 b, for example, a list image isgenerated from thumbnail images or the like of each panorama image filePDr read from the recording device 106, and a process is performed whichsets this to display image data PDdsp as a list display.

In the scroll display process 403 d, display image data PDdsp isgenerated as a scroll display for the supplied panorama image data PD.

In the projection display process 403 e, a display projection process isperformed for the supplied panorama image data PD, and display imagedata PDdsp is generated as a projection display.

The display image data PDdsp generated in any of these processes issupplied to the display section 104 and a screen display is performed.

FIG. 27 shows a process example of the control section 103 (the displayimage generation section 20) corresponding to the display modetransitions of FIG. 25.

When reproduction and display is stated in accordance with a useroperation or the like, in step F201, the display image generationsection 20 performs the processes of a single view display mode. Forexample, the display image generation section 20 reads the latestpanorama image file PDr from the recording device 106, generates displayimage data PDdsp by performing the single view display process 403 b,and supplies it to the display section 104.

In the period of a single view display mode, the display imagegeneration section 20 monitors a trigger of a mode transition or adisplay end in steps F202, F203 and F206 (the display system selectionprocess 403 a).

If the display image generation section 20 detects an end trigger, thereproduction and display operation ends from step F202.

The display image generation section 20 proceeds from step F203 to F204when an operation of a list display mode is detected, and the listdisplay process 403 c is performed. For example, a list image isgenerated by reading thumbnail images of each panorama image file PDrfrom the recording device 106, and is supplied to the display section104 as display image data PDdsp.

In step F205, the display image generation section 20 monitors a triggerof a list display end (the display system selection process 403 a). Inthe case of a list display end, the single view display process 403 b isperformed by returning to step F201. For example, display image dataPDdsp is generated as a single view display, for a panorama imageselected in the list, and is supplied to the display section 104.

When an enlarged display (a scroll display or a projection display) isspecified in a single view display state, the display image generationsection 20 proceeds from step F206 to F207, and first confirms thepanorama processing information Ip. That is, it is determined whether ornot the image presently in a single view display is a full circumferencepanorama image. In this case, the display image generation section 20can determine whether or not the panorama image data PD is a fullcircumference panorama image, by any of panorama mode type information,360° viewing angle determination information and both end alignmentinformation in the panorama processing information Ip.

In the case where it is a full circumference panorama image, the displayimage generation section 20 proceeds from step F208 to F211, generatesdisplay image data PDdsp, by performing the projection display process403 e, and supplies it to the display section 104.

In step F212, the display image generation section 20 monitors an end ofa projection display mode, and returns to the single view displayprocess 403 b of step F201 if it is finished.

In the case where it is not a full circumference panorama image, thedisplay image generation section 20 proceeds from step F208 to F209,generates display image data PDdsp, by performing the scroll displayprocess 403 d, and supplies it to the display section 104.

In step F212, the display image generation section 20 monitors an end ofa projection display mode, and returns to the single view displayprocess 403 b of step F201 if it is finished.

Various types of displays are executed in the mode transitions shown inFIG. 25 by the above described processes.

That is, in the imaging apparatus 50, in the case where the panoramaimage data PD to be a display target is determined to be a fullcircumference panorama image, the control section 103 (the display imagegeneration section 20) generates display image data PDdsp by performinga display projection process, and causes a projection display to beexecuted.

In the case where it is determined not to be a full circumferencepanorama image, display image data PDdsp is generated in a scrolldisplay process of only sequentially cutting panorama image data PD froma projection surface at the time of combining, without performing adisplay projection process, and a scroll display is executed.

Note that, as a modified example of this process, in the case where itis determined not to be a full circumference panorama image, a singleview display process may be performed, as a display process which doesnot perform a display projection process. For example, it is an examplein which an enlarged display is not performed other than for a fullcircumference panorama image, even if a specification of an enlargeddisplay is made in a single view display process.

Further, in the imaging apparatus 50 as the fifth configuration exampleof this embodiment, while an example has been set in which the imageprocessing apparatuses 1 and 2 are installed in line with the abovedescribed first configuration example, an example can naturally beassumed in which the image processing apparatuses 1 and 2 are includedin line with the above described second to fourth configurationexamples.

8. Sixth Configuration Example (Application Example to a ComputerApparatus and Program)

An application example to a computer apparatus and a program will bedescribed as a sixth configuration example. The processes as the abovedescribed image processing apparatuses 1 and 2 can be executed byhardware, or can be executed by software.

The program of an embodiment is a program, for example, which causes acalculation processing apparatus such as a Central Processing Unit (CPU)or a Digital Signal Processor (DSP) to execute the processes shown inthe above described embodiment.

That is, a program which implements the image processing apparatus 1 isa program which causes a calculation processing apparatus to execute aprocess, in the case where panorama image data generated by using aplurality of frame image data obtained by an imaging operation whiledisplacing an imaging direction is determined to be a full circumferencepanorama image, which associates the panorama image data withinformation which shows that it is a full circumference panorama image.

Specifically, this program may be a program which causes a calculationprocessing apparatus to execute the processes shown in FIG. 12A, FIG.14A, FIG. 16A, FIG. 18A or FIG. 24.

Further, a program which implements the image processing apparatus 2 isa program which causes a calculation processing apparatus to execute aprocess, in the case where panorama image data to be a display target isdetermined to be a full circumference panorama image, which generatesdisplay image data by performing a display projection process.

Specifically, this program may be a program which causes a calculationprocessing apparatus to execute the processes shown in FIG. 12B, FIG.14B, FIG. 16B, FIG. 18B or FIG. 27.

By these programs, the above described image processing apparatuses 1and 2 can be implemented by using a calculation processing apparatus.

Programs such as these can be recorded in advance to an HDD, a ROMwithin a microcomputer having a CPU or the like as a recording mediumbuilt into a device such as a computer apparatus.

Alternatively, they can be temporarily or permanently stored (recorded)in a removable recording medium such as a flexible disc, a Compact DiscRead Only Memory (CD-ROM), a Magnet Optical (MO) disc, a DigitalVersatile Disc (DVD), a Blu-ray disc (Blu-ray (registered trademark)disc), a magnetic disc, a semiconductor memory or a memory card. Such aremovable recording medium can be provided as so-called packagesoftware.

Further, other than being installed in a personal computer or the likefrom a removable recording medium, such programs can be downloaded via anetwork such as a Local Area Network (LAN) or the internet, from adownload site.

Further, if programs such as these, they will be suitable forwide-ranging provisions of the image processing apparatuses 1 and 2 ofan embodiment. For example, by downloading a program to a personalcomputer, a portable information processing apparatus, a mobile phoneunit, a game device, a video device, a Personal Digital Assistant (PDA)or the like, this portable information processing apparatus or the likecan be set to the image processing apparatuses 1 and 2. For example, ina computer apparatus such as that shown in FIG. 28, processes the sameas those of the image processing apparatuses 1 and 2 of an embodimentcan be executed.

In FIG. 28, A CPU 71 of a computer apparatus 70 executes various typesof processes in accordance with programs stored in a ROM 72 or programsloaded from a storage section 78 to a RAM 73. Further, necessary data orthe like is also arbitrarily stored in the RAM 73 upon the CPU 71executing various types of processes.

The CPU 71, the ROM 72 and the RAM 73 are mutually connected via a bus74. Further, an input/output interface 75 is also connected to this bus74.

An input section 76 constituted by a keyboard, a mouse or the like, anoutput section 77 constituted by a display constituted by a Cathode RayTube (CRT), LCD, organic EL panel or the like, and a speaker or thelike, a storage section 78 constituted by a hard disc or the like, and acommunication section 79 constituted by a modem or the like, areconnected to the input/output interface 75. The communication section 79performs a communication process via a network which includes theinternet.

Further, a drive 80 is connected to the input/output interface 75 asnecessary, a removable media 81 such as a magnetic disc, an opticaldisk, a magneto-optical disc or a semiconductor memory is arbitrarilyincluded, and computer programs read from these are installed in thestorage section 78 as necessary.

In the case where the processes of the above described image processingapparatuses 1 and 2 are executed by software, programs constituting thissoftware are installed from a network or a recording medium.

For example, as shown in FIG. 28, this recording medium is constitutedby the removable media 81, separate from the apparatus body, which isdistributed for delivering programs to a user. For example, theremovable disc 81 is constituted by a magnetic disc (including aflexible disc), an optical disc (including a Blu-ray disc, a CD-ROM or aDVD), a magneto-optical disc (including a Mini Disc (MD)) or asemiconductor memory to which programs are recorded.

Alternatively, the recording medium is constituted by the ROM 72, a harddisc included in the storage section 78 or the like, to which programsare recorded which are delivered to a user in a state built into theapparatus body in advance.

A computer apparatus 70 such as this implements the functions of theimage processing apparatus 1, based on programs by the CPU 71, wheninputting frame image data FM #0 to FM #(n−1) of n frames for panoramaimage generation, by a reception operation by the communication section79, a reproduction operation by the drive 80 (removable media 81) or therecording section 78 or the like, and executes processes as the abovedescribed association section 11.

In this way, one panorama image data is generated from the frame imagedata FM #0 to FM #(n−1) of the n input frames.

Further, the functions of the image processing apparatus 2 areimplemented based on the programs by the CPU 71, for a display of apanorama image, and processes are executed as the above describeddisplay image generation section 20.

9. Modified Example

While an embodiment has been described heretofore, various types ofmodified examples can be considered for the image processing apparatusesof the present disclosure.

The first configuration example to the fourth configuration example canbe adopted in a combined manner.

For example, adding a dedicated program with metadata as an associationprocess, by combining the first configuration example and the secondconfiguration example, performing processes of making a dedicated formatwith metadata as an association process, by combining the firstconfiguration example and the fourth configuration example or the likecan be considered.

While an example which includes the functions of both the associationsection 11 and the display image generation section 20 is set as theimaging apparatus 50 of the fifth configuration example, an example canalso be considered, for example, which includes only the functions ofone of the association section 11 and the display image generationsection 20.

While there is no target of a projection display for panorama image dataPD which is not a full circumference panorama image in an embodiment, aprojection display may be performed by a specific operation. However, inthis case, since an accurate three-dimensional model is not able to bebuilt, there is the possibility for negative effects such as theoccurrence of distortions to occur in a projection display image, and sothere is the idea of allowing this as a special mode.

Besides being built into the above described imaging apparatus 50 or thecomputer apparatus 70, the image processing apparatus of the presentdisclosure is also useful for being built into a mobile phone unit, gamemachine or video machine having an imaging function, or a mobile phoneunit, a game device, a video device or an information processingapparatus not having an imaging function but having a function whichinputs frame image data.

The present technology may also be configured as below.

(1)

An image processing apparatus including:

an association section configured to, in a case where panorama imagedata generated by using a plurality of frame image data obtained by animaging operation while displacing an imaging direction is determined tobe a full circumference panorama image, associate the panorama imagedata with information showing that the panorama image data is the fullcircumference panorama image.

(2)

The image processing apparatus according to (1),

wherein the panorama image data is determined to be the fullcircumference panorama image in a case where the panorama image data hasa 360° viewing angle.

(3)

The image processing apparatus according to (1),

wherein the panorama image data is determined to be the fullcircumference panorama image in a case where the panorama image data hasa 360° viewing angle, and an alignment process of both image ends isperformed.

(4)

The image processing apparatus according to (1),

wherein the panorama image data is determined to be the fullcircumference panorama image in a case where the panorama image data isobtained by a 360° panorama imaging mode operation in which the panoramaimage data is generated by using a plurality of frame image dataobtained by an imaging operation while displacing an imaging directionby approximately 360°.

(5)

The image processing apparatus according to any of (1) to (4),

wherein the association section associates metadata, as the informationshowing that the panorama image data is the full circumference panoramaimage, with the panorama image data.

(6)

The image processing apparatus according to any of (1) to (4),

wherein the association section associates a display processing programused for a display of the full circumference panorama image, as theinformation showing that the panorama image data is the fullcircumference panorama image, with the panorama image data.

(7)

The image processing apparatus according to any of (1) to (4), whereinthe association section sets the panorama image data itself, asinformation showing that the panorama image data is the fullcircumference panorama image, to data of a specific format.

(8)

The image processing apparatus according to any of (1) to (4),

wherein the association section associates a data file which includesthe panorama image data with a data file which includes the informationshowing that the panorama image data is the full circumference panoramaimage.

(9)

The image processing apparatus according to (5),

wherein information showing a type of projection surface of frame imagedata at the time of panorama image generation is included in themetadata.

(10)

The image processing apparatus according to (9),

wherein the information showing a type of projection surface isinformation showing either a single plane surface, cylindrical surface,spherical surface or cube as a type of curved or planar projectionsurface.

(11)

The image processing apparatus according to any of (5), (9), and (10),

wherein information of a 360° viewing angle determination result of thepanorama image data is included in the metadata.

(12)

The image processing apparatus according to any of (5), (9), (10), and(11),

wherein information showing a result of an alignment process of bothimage ends of the panorama image data is included in the metadata.

(13)

The image processing apparatus according to any of (5), (9), (10), (11),and (12),

wherein information showing whether or not the panorama image data isobtained by a 360° panorama imaging mode operation, in which thepanorama image data is generated by using a plurality of frame imagedata obtained by an imaging operation while displacing an imagingdirection by approximately 360°, is included in the metadata.

(14)

The image processing apparatus according to any of (5), (9), (10), (11),(12), and (13),

wherein information of a horizontal viewing angle and a vertical viewingangle of the panorama image data is included in the metadata.

(15)

The image processing apparatus according to any of (1) to (14), furtherincluding:

a panorama combination section configured to generate panorama imagedata by combining the plurality of frame image data obtained by theimaging operation while displacing the imaging direction.

The invention claimed is:
 1. An image processing apparatus comprising:an image device configured to acquire a plurality of frames based ondisplacement of the image device in an imaging direction during animaging operation; and circuitry configured to: generate a panoramaimage based on the plurality of frames, at least one of the plurality offrames including a seam portion that is not used in forming the panoramaimage; associate data related to the plurality of frames with panoramaprocessing information showing that the data is related to a fullcircumference panorama image; determine whether or not the panoramaimage is the full circumference panorama image based on the panoramaprocessing information; and perform a display projection process togenerate panoramic display image data when a determination resultindicates that the panorama image is the full circumference panoramaimage and another display process which is different from the displayprojection process data when the determination result indicates that thepanorama image is not the full circumference panorama image, and inwhich the circuitry is configured to use a horizontal viewing angle anda vertical viewing angle to control a display range in a scrollingoperation.
 2. The image processing apparatus according to claim 1,wherein the data related to the plurality of frames is the panoramaimage.
 3. The image processing apparatus according to claim 1, whereinthe panorama image is determined to be the full circumference panoramaimage in a case where the panorama image is obtained by a 360° panoramaimaging mode operation.
 4. The image processing apparatus according toclaim 3, wherein the 360° panorama imaging mode operation in which thepanorama image is generated by using the plurality of frames obtained bythe imaging operation while displacing the imaging direction byapproximately 360°.
 5. The image processing apparatus according to claim1, wherein the circuitry is further configured to: control a displayprocess through a single view mode of the panorama image in which anentirety of the panorama image is displayed in one image.
 6. The imageprocessing apparatus according to claim 5, wherein the circuitry furtherconfigured to; control the display process in accordance with a userinput with respect to the single view mode.
 7. The image processingapparatus according to claim 1, wherein the another display process is ascroll process.
 8. An image processing method comprising: acquiring aplurality of frames based on displacement of an image device in animaging direction during an imaging operation; forming a panorama imageusing the acquired plurality of frames, at least one of the acquiredplurality of frames including a seam portion that is not used in formingthe panorama image; associating data related to the acquired pluralityof frames with panorama processing information showing that the data isrelated to a full circumference panorama image; determining whether ornot the panorama image is the full circumference panorama image based onthe panorama processing information; and performing a display projectionprocess to generate panoramic display image data when a determinationresult indicates that the panorama image is the full circumferencepanorama image and another display process which is different from thedisplay projection process when the determination result indicates thatthe panorama image is not the full circumference panorama image;controlling a display range as part of a scrolling operation based on ahorizontal viewing angle and a vertical viewing angle.
 9. The imageprocessing method according to claim 8 further comprising, controlling adisplay process to display the full circumference panorama image as oneimage.
 10. The image processing method according to claim 8 furthercomprising, wherein the another display process is a scroll process. 11.The image processing method according to claim 8, wherein the datarelated to the plurality of frames is the panorama image.
 12. The imageprocessing method according to claim 11 wherein performing the displayprojection process further comprises, projecting the data onto a planarsurface to derive display image data.